Algorithms for Independent Task Placement and Their Tuning in Demand-Driven Ray Tracing

Bratislava, January 2005Tomas Plachetka, University of Bratislava / Paderborn / Bristol 1

Algorithms for Independent Task Placement and Their Tuning in Demand-Driven Ray Tracing

Algorithms for Independent Task Placement

and Their Tuning in the Context of Demand-Driven Parallel Ray Tracing


Overview

• Demand-driven process farm• Abstraction from assignment mechanism, examples of trade-offs• Formal problem definition• Analysis of chunking and factoring strategies• Experiments with chunking and factoring, with manual tuning

of parameters• Tool for the prediction of efficiency for process farms

(more precisely, “post-prediction”) • If the parameters are tuned then the choice of assignment

strategy does not significantly influence efficiency of parallel

computation in the context of parallel ray tracing on

contemporary machines with an “everyday” input, “everyday”

quality settings etc.


Demand-Driven Process Farm

Eye(camera)

Output image

MASTER

1 2

3 4

result

LOADBALANCER

job req

job

WORKERS

How many tasks should the LOADBALANCER process assign in one job?

(Badouel et al. [1994] suggest 9 pixels in one job, Freisleben et al. [1998]

suggest 4096 pixels in one job… Where does this difference come from?)


Abstraction from assignment mechanism

LB

WORKER1 WORKER2 WORKER3 WORKER4

Message passing network (send, receive)

WORKER1 WORKER2 WORKER3 WORKER4

Shared memory (central queue, locking)

The assignment mechanism is not important. Assignment of one job costs constant time L, no matter how many tasks are in the job which is assigned to a worker.


Trade-offs for chunking (which assigns fixed-size jobs)Largest jobs (for 2 workers)

1 2

Smallest jobs (1 job=1 task)

Problem: imbalance Problem: many messages

How large should the jobs be?

(so that the parallel time is minimal!)

The job “shape” is irrelevant, only the size is important


Problem definitionGiven:N nr. of worker processes, all equally fastW nr. of tasks, independent on each other (not even spatially coherent)L latency; i.e. penalty for assigning 1 job to a worker (a constant time which does not depend on the number of tasks in the job or anything else)

Unknown:Task time complexities.Goal:Minimise the makespan (the parallel time required for assigning & processing of all tasks). The LOADBALANCER must make a decision as to how many tasks to pack into a job immediately after receiving a job request. (This is not quite online… note that W is constant!)

Probabilistic model average tasks’ time complexity std. dev. of tasks’ complexities

Goal: minimise expected makespan

Deterministic modelTmax maximal tasks’ time complexityTmin minimal tasks’ time complexity

Goal: minimise maximal makespan(for worst possible task arrangement)


Chunking strategy (fixed-size chunks)

LB_CHUNKING(float Tmax, int W, int N, float L)

{

int work = W;

K=???;

wait for a job request;

if (K > work)

K = work;

assign job of size K to the idle WORKER;

work = work – K;

}


Chunking, analysis

)(1 maxKTLNK

WM high

0:)('2max

NK

WLTKM

maxNT

WLKopt

N nr. of workersW nr. of tasksL latency Tmax max. task complexityUnknown: Kopt (chunk size)

The time diagram below depicts thestructure of the worst case (maximal makespan):One of the workers always gets the tasks oftime complexity Tmax. (The last extra-round is the result of integer arithmetic.)

K K

…

KK

N

L+KTmaxL+KTmax L+KTmax

L+KTmax

N

LWTL

N

WTM high

maxmax 2

N

LWT

N

WTM low

minmin


Chunking, probabilistic model

Chunking (Kruskal and Weiss)

NKNK

WL

N

WME ln2][

for large W and K and K>>log N.

K

NK

NK

WL

N

WME

ln2][

NN

WLK opt

ln

2

for K<< W / N and small sqrt(K) / N.

?optK

N

NK

WL

N

WME

2

][ ?optK

for K<< W / N and large sqrt(K) / N.


Factoring strategy, example

)1(1

,1maxNT

WK rest

1

N

KWTK

N=2T=3

Parameterisation:N nr. of workersW nr. of tasks T max. ratio of tasks’ complexities (T=Tmax/Tmin)Unknown: the job size K used for the next round

t1 sec t2 sec≤

T·

t sec t sec

≤ T·

1

)()(

N

KWtKt

Wx

4W

4

33 ≥ x = 1

Example


Factoring, analysis

)1,(memin_par_ti)(memax_seq_ti NKwK iii

wi denotes the number of yet unassigned tasks after round i. Obviously, the larger the assigned job sizes Ki are, the smaller is the assignment overhead. Hence, we want:

)1,(memin_par_ti)(memax_seq_ti NKwK iii

Left-hand side:

max)(memax_seq_ti TKLK ii Right-hand side, simplified (the assignment latency is only counted once):

1

)()1,(memin_par_ti min

N

TKwLNKw ii

ii

Note that this simplification ignores the assignment latency. Solving the simplified equation above yields (we denote T=Tmax/Tmin)

)1(1

NT

wK isimplei


Factoring (simplified), analysis

This work remaining after round i

1

)1(11

i

i NT

NWw

yields WNr NTN /log1 ))1(1/(1


Factoring (simplified), analysis

Makespan, upper-bound )1()1/(max rLNWTM high

Makespan, lower-bound LrNWTM low /min


Factoring, probabilistic model

Factoring (Flynn-Hummel)

i

ii

ii

iii

Nx

wK

w

Nx

w

Nx

NKww

Ww

22

1

22

1

1

1

2

11

wi is the rest of work at the beginning of round i

1 / (N xi) is the division factor

Ki is the chunk size for round i

In their experiments [1991, 1995], the authors did not attempt to estimate the covariance σ/µ. They used constant division factor 1/(2N) (this means xi =2).


Experiments: data


Experiments: setting

(Machine: hpcLine in PC2)

(Application: parallel ray tracing with “everyday setting”)

Given:N=1…128 nr. of worker processes, all equally fastW=720*576 nr. of tasks, independent on each otherL=0.007 latency; i.e. penalty for assigning 1 job to a worker

(a constant time which does not depend on thenumber of tasks in the job or anything else)

Estimated from measured data:Tmax=0.00226average time on one pixel (360 pixels)Tmin=0.00075 factor between the times on atomic jobs of

size 360 pixels (i.e. T=ca. 3)


Experiments: empirical optimal chunk size (90 workers)

pixelsAK FACTORINGopt

CHUNKINGopt 36045ˆˆ


Experiments: chunking efficiency (K=360)


Experiments: factoring efficiency (atomic_job_size=360)


Tuning of assignment strategies: estimation of future

K=f(W, N, L, Tmin, Tmax)

Suggestion: the unknown parameters Tmin and Tmax can be initially estimated.

This estimation is continually adjusted according to measured run-time statistics. (The estimation of remaining time needed for copying files in Windows uses a similar approach.)

The optimal chunk size for chunking and factoring strategies depends on parameters which are unknown. (However, all these parameters are known when the computation finishes; this is what we call “post-prediction”!)


Conclusions• Farming yields almost a linear speedup (efficiency 95% with 128 workers) for parallel ray tracing (POV||Ray) on a fairly complex “everyday” scene.

• Trivial chunking algorithm with optimal chunk size does not perform worse than a theoretically better factoring algorithm with optimal chunk size; for the particular machine, particular nr. of processors, particular input, particular quality settings, particular room temperature etc. used during experiments.

• Efficiency of chunking/factoring can be predicted (or at least “post-predicted”) for a particular machine, particular nr. of processors, particular input, particular room temperature etc.

• In experiments with process farming, parameters W (nr of tasks), N (nr of workers), L (latency), Tmin and Tmax (min/max tasks’ or jobs’ time complexities)

must be reported. Reporting only some of these parameters is insufficient for drawing conclusions from experiments with process farming (e.g. chunking).

• The parameters specific to chunking/factoring can (and must) be tuned automatically in run-time.

Documents

Algorithms for Independent Task Placement and Their Tuning in Demand-Driven Ray Tracing