Scaling Area Under a Curve

Why do parallelism?

• Speedup – solve a problem faster.• Accuracy – solve a problem better.• Scaling – solve a bigger problem.

Problem 1: communication overhead

• Parallel algorithms that employ distributed memory have a certain amount of communication and a certain amount of computation.

• As more processes are added, the communication overhead increases.

• At some number of processes, communication overhead will cause the algorithm to take longer in parallel than serially.

Problem 2: Amdahl’s law

• Speedup is limited by the algorithm’s critical section – the part of the algorithm that cannot be parallelized.

• An algorithm will never be faster than its critical section.

Speedup =

where P = the proportion of the program that can be made parallel1 – P = the proportion of the program that cannot be made

parallelN = the number of processors

€

1

1 −P +P

N

Strong scaling

• Strong scaling – increasing the number of processes but keeping the problem size constant.

The solution

• Don’t solve a problem faster, instead solve a bigger problem.

Gustafson’s law

• A bigger problem solved with more processors can be solved in the same amount of time it takes to solve a smaller problem with fewer processors.

Speedup(N) = N – (1 – P) * (N – 1)where

N = the number of processors(1 – P) = the proportion of the program that cannot

be made parallel

Weak scaling

• Weak scaling – increasing the problem size as the number of processes increases.