Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Evaluating the
Parallel Performance
of a Heterogeneous System
Elizabeth Post
Hendrik Goosenformerly ofDepartment of Computer ScienceUniversity of Cape Town, South Africa
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Introduction
• Why measure and evaluate parallel performance?
• What to measure• How to evaluate• Why speedup and efficiency are not appropriate• Alternate methods of performance evaluation
– Power weight– Linear speed– Linear efficiency
• Conclusions
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Why measure parallel performance?
• Increasing use of parallel processing– Clusters, such as Beowulfs– Networks of Workstations (NOWs)
• Scalability– Does performance improve as more processors are
added?– Will performance continue to improve as more
processors are added?• Efficiency
– Is the best possible performance being achieved?– Where and how can performance be improved?– Is the best algorithm being used?
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Why use a real application to measure performance?
• Real users want to know how long real applications take
• Measures such as:– MIPS– MFLOPS– Kernels– Vendor-tuned benchmarks
do not take into account factors such as:– input/output, – communication time, – memory needs and usage, – idle time etc.
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Background
• Climatology application– Cloud radiation simulation to model and measure
• reflectivity,• transmissivity, and• absorptivity
of a heterogeneous strato-cumulus cloud deck.
• Environment– Network of Unix workstations,
• five different models of Silicon Graphics and Sun workstations • with varying CPU performance and memory capacity,
– connected by a 10Mbit Ethernet network.
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Serial times for processorsCPU, System and Elapsed times
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Parallel performance -What to measure?
• Elapsed time
• CPU time
• System time
• Communication time
• Idle time
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Parallel performance -How to evaluate?
• Elapsed time graphs
• Speedup
• Efficiency
• Power weight
• Linear speed
• Linear efficiency
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Heterogeneous Parallel groupCPU, System and Elapsed times
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Speedup
Speedup is:
the ratio of the serial time taken on one processor compared to the parallel time on all processors
Elapsed time on 1 processor
Elapsed time on n processors
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Speedup – how to calculate it?
• Which single processor elapsed time should be used?– Elapsed time of fastest processor?– Elapsed time of slowest processor?– Mean elapsed time of all processors used?
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Speedup vs Perfect Speedup(calculated with means)
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Efficiency
Efficiency is:
the ratio of the speedup divided by the number of processors used
Speedup for n processors
n
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Efficiency(calculated with means)
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Speedup and efficiency not appropriate
• Conclusion that speedup and efficiency are not appropriate for evaluating parallel performance on a heterogeneous system, and even have their limitations on a homogeneous system.
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Power weight(Zhang et al)
Power weight is:ratio of performance of each processor as compared to performance of the fastest processor
Elapsed time for fastest processor
Elapsed time for nth processor
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Linear Speed (Crowl)
Linear Speed is:
the amount of work done in unit time
1Total elapsed time
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Linear Speed
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Linear Efficiency (Post and Goosen)
Linear Efficiency is:
ratio of work done in parallel application compared to potential amount of work that could be done by all processors
Linear speed for parallel application on n processorsSum of single processor linear speeds for n processors
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Linear Efficiency
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Advantages of linear speed and linear efficiency
• Linear speed for each machine is independent of all other machines
• Linear speeds can be recalculated dynamically as workloads vary and used to determine each machine’s current performance compared to its serial capacity.
• Dynamic calculation of linear speeds and linear efficiency can be used for dynamic load-balancing algorithms.
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land
Conclusions
• Important to measure overall elapsed time, as well as components of CPU, communication, idle time etc.
• Speedup and efficiency are not appropriate for evaluating parallel performance, especially for heterogeneous systems
• Linear speed and linear efficiency provide useful ways of evaluating parallel performance for both heterogeneous and homogeneous systems.
• Linear speed and linear efficiency can be calculated dynamically and be used in dynamic load balancing algorithms.
Lin
coln
Uni
vers
ity
Can
terb
ury
New
Zea
land