Data-Level Parallelism: SIMD, Vector & GPUs

data-level parallelism

Suppose a teacher hands every one of thirty students the same one-line instruction — 'add 5 to your number' — and they all do it at the same moment, each to their own number. The instruction is identical for everyone; only the data differs. That picture is data-level parallelism: one operation applied to a whole pile of data elements at once, instead of looping over them one at a time. It is a different flavour of parallelism from running thirty different programs side by side.

In a computer, data-level parallelism shows up wherever a program does the same arithmetic across a large, regular array of numbers. Brightening an image means add the same value to millions of pixels. Mixing two audio tracks means add matching samples in two long arrays. Multiplying matrices, simulating physics on a grid, or training a neural network all boil down to multiply-and-add applied to huge tidy arrays. A normal CPU would write a loop and process one element per iteration; a data-parallel machine processes many elements with one instruction, because the work on element i never depends on the work on element j.

Data-level parallelism is one branch of a famous split. Flynn's taxonomy calls a machine that runs one instruction stream over many data items SIMD (single instruction, multiple data). The other big branch, thread-level parallelism, runs many independent instruction streams (one per core) and is what multicore chips exploit. Both are real, and modern systems use both — but they want different code. DLP shines on regular, predictable, independent element-wise work and is the foundation under SIMD instructions, vector processors, and GPUs; it stumbles badly the moment the work becomes irregular or each element's result depends on its neighbour's.

Brightening a photo: for one million pixels, add 20 to each. A scalar loop runs the add a million times. A data-parallel unit that holds 8 pixels per register runs the same add 125,000 times, each handling 8 pixels — roughly 8x fewer instructions for the same result, because no pixel's brightness depends on another's.

Element-wise independent work is the sweet spot for data-level parallelism: one instruction, many elements.

Data-level parallelism is not a free upgrade. If element i's result feeds element i+1 (a dependence), or the elements need different operations, the parallelism collapses — and forcing it on irregular work often runs slower than a plain loop.

Also called
DLPdata parallelism資料平行