Limitations of parallel processing and Amdahl's law
There are limitations of parallel processing that you must consider when developing parallel applications.
For example, if your application parallelizes perfectly, executing the application on ten processors makes it run ten times faster. However, applications rarely parallelize perfectly because part of the application is serial. This serial component imposes a limit on the amount of parallelization the application can use.
Amdahl's law describes the maximum speedup that parallel processing can achieve.
The formula for Amdahl's law is shown in the following figure where the terms in the equation are:
|S||Fraction of the application that is serial.|
|P||Fraction of the application that is parallelizable.|
|N||Number of processors.|
Figure 2-4 Formula for Amdahl's law
The following figure shows the speedup that different numbers of processors provide for applications with different serial components.
Figure 2-5 Speedup for an application with different serial components
The biggest speedups are achieved with relatively small numbers of processors. However, as the number of processors rises, the per-processor gains are reduced.
You cannot avoid Amdahl's law in your application but you can reduce the impact.
For high performance with a large number of processors, the application must have a very small serial component. These sorts of applications are said to be embarrassingly parallel.