data parallelism

Imagine a thousand exam papers that all need the same grading rubric applied. The fastest way to grade them is to split the pile into ten stacks and give one stack to each of ten teaching assistants. Every assistant does exactly the same job — apply the rubric — just to different papers. Data parallelism is this idea inside a program: you take one big collection of data, cut it into pieces, and run the same operation on each piece at the same time, one piece per thread or core.

Concretely, data parallelism (also called the data-decomposition view) divides the data and replicates the work. If you want to add 1 to every element of an array of one million numbers, you can give the first quarter-million elements to thread 0, the next quarter to thread 1, and so on across four cores; each thread runs the identical add loop on its own slice. The operation is the same everywhere — only the data each thread touches differs. This is the natural fit whenever you have a large, uniform dataset and an operation that applies independently to each element, which is exactly why graphics processors, which run the same shader over millions of pixels, are built around it.

The catch is that data parallelism shines only when the pieces are truly independent — when computing one slice does not need the result of another. The moment elements depend on their neighbors (as in many physics simulations), the threads must communicate or synchronize, and the clean speed-up erodes. It is also limited by Amdahl's law: any setup, any final combining step, and any uneven slice sizes (load imbalance) all eat into the gain. Contrast it with task parallelism, where the threads do different jobs rather than the same job on different data.