Where one-operation-many-data earns its keep
The previous guide planted the seed: data-level parallelism means the very same operation applied to a whole array of values, with no dependencies between elements. Before we build the machines, let us be honest about where that pattern actually shows up — because it is the precondition for everything in this guide. It lives in multimedia (add the same brightness to every pixel, mix the same gain into every audio sample), in scientific computing (multiply two vectors, step every cell of a simulation grid), and in machine learning (a matrix multiply is millions of independent multiply-add pairs). The common shape is a loop whose iterations do not talk to one another.
When that shape is present, doing the elements one at a time is wasteful: you pay the full overhead of fetching and decoding an instruction just to add a single pair of numbers, then do it again, and again. SIMD — Single Instruction, Multiple Data — fixes that by issuing one instruction that processes many elements in one go. You amortize the instruction's overhead across the whole bundle, and the hardware adds, say, eight pairs in parallel where a scalar machine would have run eight separate adds. That is the whole pitch, and it is why this style of parallelism is cheap to add to a CPU and cheaper still per result.
The vector processor: registers, lanes, and how memory feeds them
The classic embodiment is the vector processor. Where an ordinary register file holds one scalar per register, a vector register holds a whole row of elements — picture a register that is not a single box but a tray of, say, 8 or 16 or 64 slots. A single vector instruction like 'vector-add' streams that whole tray through the adder. The hardware is organized into lanes: parallel copies of the arithmetic unit, each chewing on one slot of the tray. Eight lanes means eight additions happen in the same clock cycle, side by side, like eight cashiers ringing up eight customers at once instead of one cashier serving a queue.
There is a beautiful economy here. One fetch, one decode, one issue — then a long burst of useful arithmetic. The control overhead that dominates scalar loops is spread thin, and the processor knows in advance exactly how many elements are coming, so it can keep the lanes and the pipeline fed without the per-iteration branch overhead of a scalar loop. A genuine vector machine also carries a vector length register, so one instruction can handle a tray that is only partly full — the leftover bit at the end of an array that does not divide evenly by the lane count.
But the lanes are only as fast as memory can fill them. The simplest case is a contiguous block — elements sitting one after another, loaded in one efficient sweep. Real data is messier, so vector machines support two awkward access patterns. Strided access grabs every k-th element (every third value, say, when reading one colour channel out of interleaved RGB pixels). Gather/scatter reads or writes elements at a list of arbitrary addresses given by an index array — flexible, but each element may sit in a different cache line, so it can be far slower than a clean contiguous load. The honest rule: the more scattered your data, the less of SIMD's promise you actually collect.
SIMD extensions: SSE, AVX, NEON — and the auto-vectorizer that tries for you
Few people buy a pure vector supercomputer today. Instead, vector ideas were bolted onto ordinary CPUs as a SIMD extension: a set of extra-wide registers and instructions added to the existing instruction set. On Intel and AMD you meet SSE (128-bit registers, four 32-bit floats at once) and its successors AVX and AVX-512 (256- and 512-bit registers, eight or sixteen floats at once). On Arm chips — your phone, a Mac — the equivalent is NEON (and newer SVE). Same idea everywhere: one register holds a small fixed pack of values, and one instruction operates on the whole pack. These are part of the hardware-software interface — they widen the ISA contract so software can ask for parallel arithmetic by name.
So who actually emits these instructions? Ideally the compiler, through auto-vectorization: you write an ordinary scalar loop, and the auto-vectorizer recognizes the pattern and rewrites it to process several elements per SIMD instruction. When it works it is magic — your plain C loop quietly runs four or eight times wider with no source changes. But it is fragile, and being honest about its limits is the point of this section. The compiler must prove the iterations are truly independent; the moment it cannot, it gives up and emits slow scalar code, often without telling you.
// vectorizes cleanly: independent, contiguous, countable
for (i = 0; i < n; i++)
c[i] = a[i] + b[i]; // 8 adds per AVX instruction
// will NOT vectorize: each element depends on the previous
for (i = 1; i < n; i++)
a[i] = a[i-1] + a[i]; // loop-carried dependence
// may NOT vectorize: aliasing unknown (do a and b overlap?)
void f(float *a, float *b, int n) {
for (i = 0; i < n; i++) a[i] = b[i] * 2.0f;
}The GPU: a thousand interns instead of one genius
Push the SIMD idea to its extreme and you arrive at the GPU. A graphics processing unit was born to colour millions of pixels, every one an independent little calculation — exactly the no-dependency shape from the first section. So instead of a few wide CPU cores, a GPU packs thousands of small, simple arithmetic units. The analogy that sticks: a CPU is one brilliant genius who can do anything, including tricky serial reasoning, very fast; a GPU is a thousand interns, each only able to do one tiny sum, but a thousand of them at once. For the right problem the thousand interns crush the lone genius.
This makes the GPU a throughput processor: it does not try to finish any single task quickly, it tries to finish a vast pile of tasks per second. A throughput machine tolerates long memory latency by simply having so much independent work queued up that, whenever one group of work stalls waiting for memory, the hardware instantly switches to another ready group. Where a CPU spends its transistor budget on caches and clever out-of-order machinery to make one stream fast, the GPU spends it on sheer arithmetic width and many parallel work-groups to keep all those units busy. Two different bets on where to put the silicon.
You do not write a GPU program the way you write a CPU loop. The programming model (CUDA on NVIDIA, and similar models elsewhere) asks you to write one short function — a 'kernel' — describing what to do for a single element, and then launch it across thousands of elements at once. The hardware spreads those thousands of copies over its many units. It is a lovely fit for the data-parallel shape, but it forces you to express your problem as 'the same little function, run independently a huge number of times'. Problems that genuinely have that structure soar; problems that do not are a poor fit no matter how you squint.
CPU versus GPU, and the roofline that bounds them both
Here is the honest heart of the matter, and the misconception this guide most wants to kill: a GPU is not just a faster CPU. It is a different bet. The GPU is fantastic at wide, regular, independent arithmetic — and genuinely poor at serial, branchy, latency-sensitive code, the kind with lots of if-statements, hard-to-predict control flow, and chains where each step needs the previous one's answer right now. That serial code is exactly what a CPU's out-of-order execution and branch prediction are built to chew through. Hand such code to a GPU and most of those thousand interns sit idle while a few crawl through a tangle they were never designed for.
Whether either chip even reaches its peak depends on a single, clarifying picture called the roofline model. Plot achievable performance against arithmetic intensity — the ratio of useful arithmetic operations to bytes moved from memory. At low intensity (few flops per byte, like adding two big arrays once) you are memory-bound: a sloped 'roof' set by memory bandwidth caps you, and a faster arithmetic unit buys nothing because you are starved for data. At high intensity (many flops per byte, like a dense matrix multiply that reuses each loaded value many times) you hit the flat 'roof' of peak compute. The model tells you, before you optimize, which wall you are actually pressed against.
Two further honest caveats round this out. First, getting work onto a GPU means copying data across a relatively narrow link from main memory to the GPU's own memory; for a small job that transfer can cost more than it saves, so the GPU only wins when the arithmetic is big enough to drown out the copy. Second, Amdahl's law from the multicore rung still rules: if even a small slice of your program is stubbornly serial, that slice caps your total speedup no matter how many lanes or interns you throw at the parallel part. Massive parallelism is a sharp tool for one specific shape of problem — not a universal faster button.