Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Evaluating the

Parallel Performance

of a Heterogeneous System

Elizabeth Post

Hendrik Goosenformerly ofDepartment of Computer ScienceUniversity of Cape Town, South Africa

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Introduction

• Why measure and evaluate parallel performance?

• What to measure• How to evaluate• Why speedup and efficiency are not appropriate• Alternate methods of performance evaluation

– Power weight– Linear speed– Linear efficiency

• Conclusions

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Why measure parallel performance?

• Increasing use of parallel processing– Clusters, such as Beowulfs– Networks of Workstations (NOWs)

• Scalability– Does performance improve as more processors are

added?– Will performance continue to improve as more

processors are added?• Efficiency

– Is the best possible performance being achieved?– Where and how can performance be improved?– Is the best algorithm being used?

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Why use a real application to measure performance?

• Real users want to know how long real applications take

• Measures such as:– MIPS– MFLOPS– Kernels– Vendor-tuned benchmarks

do not take into account factors such as:– input/output, – communication time, – memory needs and usage, – idle time etc.

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Background

• Climatology application– Cloud radiation simulation to model and measure

• reflectivity,• transmissivity, and• absorptivity

of a heterogeneous strato-cumulus cloud deck.

• Environment– Network of Unix workstations,

• five different models of Silicon Graphics and Sun workstations • with varying CPU performance and memory capacity,

– connected by a 10Mbit Ethernet network.

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Serial times for processorsCPU, System and Elapsed times

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Parallel performance -What to measure?

• Elapsed time

• CPU time

• System time

• Communication time

• Idle time

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Parallel performance -How to evaluate?

• Elapsed time graphs

• Speedup

• Efficiency

• Power weight

• Linear speed

• Linear efficiency

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Heterogeneous Parallel groupCPU, System and Elapsed times

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Speedup

Speedup is:

the ratio of the serial time taken on one processor compared to the parallel time on all processors

Elapsed time on 1 processor

Elapsed time on n processors

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Speedup – how to calculate it?

• Which single processor elapsed time should be used?– Elapsed time of fastest processor?– Elapsed time of slowest processor?– Mean elapsed time of all processors used?

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Speedup vs Perfect Speedup(calculated with means)

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Efficiency

Efficiency is:

the ratio of the speedup divided by the number of processors used

Speedup for n processors

n

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Efficiency(calculated with means)

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Speedup and efficiency not appropriate

• Conclusion that speedup and efficiency are not appropriate for evaluating parallel performance on a heterogeneous system, and even have their limitations on a homogeneous system.

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Power weight(Zhang et al)

Power weight is:ratio of performance of each processor as compared to performance of the fastest processor

Elapsed time for fastest processor

Elapsed time for nth processor

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Linear Speed (Crowl)

Linear Speed is:

the amount of work done in unit time

1Total elapsed time

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Linear Speed

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Linear Efficiency (Post and Goosen)

Linear Efficiency is:

ratio of work done in parallel application compared to potential amount of work that could be done by all processors

Linear speed for parallel application on n processorsSum of single processor linear speeds for n processors

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Linear Efficiency

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Advantages of linear speed and linear efficiency

• Linear speed for each machine is independent of all other machines

• Linear speeds can be recalculated dynamically as workloads vary and used to determine each machine’s current performance compared to its serial capacity.

• Dynamic calculation of linear speeds and linear efficiency can be used for dynamic load-balancing algorithms.

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land

Conclusions

• Important to measure overall elapsed time, as well as components of CPU, communication, idle time etc.

• Speedup and efficiency are not appropriate for evaluating parallel performance, especially for heterogeneous systems

• Linear speed and linear efficiency provide useful ways of evaluating parallel performance for both heterogeneous and homogeneous systems.

• Linear speed and linear efficiency can be calculated dynamically and be used in dynamic load balancing algorithms.

Lin

coln

Uni

vers

ity

Can

terb

ury

New

Zea

land