A different axis of parallelism
Every speed-up earlier in this ladder squeezed more out of a stream of instructions. Pipelining overlapped their stages like a laundry line; superscalar and out-of-order execution launched several different instructions per cycle when their data was ready; multicore handed whole independent instruction streams to separate cores. That family is instruction-level and thread-level parallelism — different work, run together. This guide turns ninety degrees and asks a new question: what if the work is identical but the data is plural?
Picture adding two arrays element by element: `c[i] = a[i] + b[i]` for a million `i`. A plain scalar loop issues one add per element, plus loop bookkeeping, plus a branch back to the top each time. Yet every iteration runs the very same add — only the operands differ. That is wasteful: we pay to fetch and decode the same instruction a million times to do a million copies of one operation. Data-level parallelism (DLP) is the observation that one decoded operation could be broadcast across many data items at once, amortizing all that fetch-and-decode overhead.
Where does such plural data come from? Everywhere the real world arrives in bulk. Multimedia is rows of pixels and streams of audio samples treated alike; scientific computing is matrices and grids stepped through with the same formula; modern machine learning is oceans of multiply-add over weight tensors. These workloads are drowning in identical-operation-many-data structure, which is exactly why the hardware in the rest of this rung exists. DLP is not an exotic trick — it is the natural shape of the most demanding programs we run.
SIMD: one instruction, many lanes
The hardware answer is SIMD — Single Instruction, Multiple Data. In Flynn's taxonomy, a plain CPU executing one op on one operand pair is SISD; SIMD keeps the single instruction but feeds it a vector of operands. Think of a row of cashiers all handed the exact same one-line instruction — 'add your two numbers' — and a tray of number pairs slid in front of them; they all add simultaneously and a tray of sums slides out. Each cashier is a lane, and the lanes march in lockstep: same operation, same cycle, different data.
To hold that tray of operands, SIMD hardware adds wide vector registers. Where an ordinary register holds one 64-bit value, a vector register is a long bin — say 512 bits — that you can slice into lanes: eight 64-bit doubles, or sixteen 32-bit floats, or thirty-two 16-bit numbers. One vector add then performs sixteen float additions in a single instruction. Crucially this is not magic extra ALUs for free; the chip really does contain sixteen adders wired side by side. SIMD's win is not more arithmetic per second from nowhere — it is paying the fetch-and-decode cost once and reusing it across the whole lane width.
Scalar: one add per instruction
a0 + b0 = c0 (decode an 'add' ... again ... again ...)
SIMD: one add across 8 lanes of a 512-bit vector register
| a0 | a1 | a2 | a3 | a4 | a5 | a6 | a7 | <- vector reg A
+ | b0 | b1 | b2 | b3 | b4 | b5 | b6 | b7 | <- vector reg B
----+----+----+----+----+----+----+----
| c0 | c1 | c2 | c3 | c4 | c5 | c6 | c7 | <- one VADD instruction
Same opcode fetched ONCE; 8 (or 16, or 32) results per issue.
Lanes never talk to each other here -- each is its own little adder.Feeding the lanes: stride, gather, and scatter
Wide adders are useless if you cannot get the data into them. The friendly case is contiguous data: `a[0..7]` sit next to each other in memory, so one wide load grabs all eight lanes in a single trip — and because that trip pulls in a whole cache line, it rides beautifully on spatial locality. This is why SIMD and the memory hierarchy are best friends: a vector load that maps onto one cache line is the cheapest possible way to feed the lanes.
Real data is not always so tidy. Sometimes you want every fourth element (the red channel of an RGBA image), which is a strided access — regular gaps of a fixed step. Worse, sometimes the indices are themselves data, as in `c[i] = a[index[i]]`, where each lane must reach a different, unpredictable address. Hardware names these gather (collect scattered addresses into a vector register) and scatter (write a vector back out to scattered addresses). Gather/scatter make SIMD far more general, but be honest about the cost: a gather that touches eight different cache lines is dramatically slower than one contiguous load, because the lanes are no longer sharing a single trip to memory.
From extensions to auto-vectorization (and its limits)
You meet SIMD in practice as instruction-set extensions bolted onto an existing CPU. On x86 these are SSE and AVX (AVX-512 gives the 512-bit registers above); on Arm — the chip in your phone — it is NEON, with the newer SVE letting the vector length vary. Each is an ISA extension: extra opcodes and vector registers grafted onto a scalar core, sharing its pipeline and caches. Notice the binary-compatibility wrinkle this creates — a binary using AVX-512 simply will not run on an older chip that lacks those instructions, which is why software often ships multiple code paths and picks one at startup.
Who actually writes vector instructions? Ideally not you. Auto-vectorization is the compiler quietly rewriting a scalar loop into vector instructions when it can prove the rewrite is safe. The `c[i] = a[i] + b[i]` loop is the dream case: independent iterations, contiguous data, a known trip count. The compiler peels it into chunks of, say, eight, emits one SIMD add per chunk, and cleans up any leftover elements at the end. When it works, you get a big speed-up for free with the same source code.
But auto-vectorization is fragile, and honesty matters here. The compiler must prove the iterations are independent; if two array references might alias (point at overlapping memory), or the loop carries a dependency like `a[i] = a[i-1] + 1`, or it branches unpredictably inside, the compiler conservatively refuses and you silently get plain scalar code. This is why performance engineers learn to write loops the compiler can vectorize — simple bodies, no aliasing, contiguous access — or, failing that, drop down to hand-written intrinsics. SIMD is wonderful, but it is not automatic in practice; it is a contract the code must keep.
The GPU: when 'many data' becomes 'thousands'
Push DLP to its extreme and you get the graphics processing unit. Where a CPU's SIMD unit has perhaps sixteen lanes bolted to a clever, latency-hunting core, a GPU pours nearly all its silicon into thousands of simple arithmetic units. Our analogy from earlier in the ladder fits perfectly: a CPU is one brilliant chef who can improvise around any obstacle, while a GPU is a thousand interns each doing one tiny sum. For the array-add over a million elements, the thousand interns annihilate the lone genius — that is the GPU as a throughput machine.
GPUs run a cousin of SIMD that the next guides will unpack: SIMT (Single Instruction, Multiple Threads). You write what looks like ordinary code for one data element — a kernel — and the hardware runs thousands of copies of it, bundling them into groups (a warp) that execute in lockstep like SIMD lanes. It is a friendlier way to express the same DLP. We will see in the SIMT guide how this lockstep both empowers the GPU and trips it up the moment threads in a warp want to take different branches.
End with the crucial, honest contrast that the rest of this rung will sharpen. A GPU is not just a faster CPU. It wins overwhelmingly on huge, regular, independent data — but it is genuinely poor at serial, branchy, latency-sensitive code, the very thing a CPU's branch predictors and out-of-order machinery are built to excel at. Whether a workload is bandwidth-bound or compute-bound is captured by the roofline model we will meet later. The headline: throughput and latency are different goals, and no single chip wins both — which is exactly why modern systems pair a CPU and a GPU rather than choosing.