From wider lanes to a wholly different machine
In the last two guides we widened a single CPU core. We bolted SIMD lanes onto its datapath, grew its vector registers from 128 to 512 bits, and leaned on the instruction set extensions and auto-vectorization to feed one operation across many data elements at once. That is a powerful trick, but notice its ceiling: it is still one core, with one program counter, one branch predictor, one out-of-order engine, just doing each instruction on a fatter slab of data. We made the genius's hands wider. What if instead we hired a thousand ordinary people?
That is the leap a graphics processing unit makes. A modern CPU spends most of its silicon not on arithmetic but on cleverness for one thread: big caches, deep out-of-order machinery, accurate branch prediction — all to make a single chain of dependent instructions finish as soon as possible. A GPU throws almost all of that away. It fills the same area with hundreds or thousands of small, simple arithmetic units and almost no per-thread cleverness, betting that you have brought enormous amounts of independent work for them to chew through in parallel.
Where this kind of work hides
A GPU is useless unless you can actually feed it that mountain of independent work, so the first real skill is recognizing it. The historical home is graphics itself: a screen is millions of pixels, and shading each one is roughly the same little calculation with no pixel needing to wait on its neighbour. That is pure data-level parallelism — the same operation, many data, no dependencies — exactly the shape the last two guides described, just at a far larger scale than a 16-wide SIMD register could ever cover.
The same shape turns up far beyond graphics. Multiply two large matrices and every output cell is an independent dot product. Simulate physics on a grid and every cell updates from its neighbours by the same rule. Train a neural network and you are doing oceans of matrix multiplies and element-wise functions. This is why GPUs quietly became the engines of scientific computing and modern machine learning: those workloads are naturally shaped like ten thousand identical sums. When you can phrase your problem that way, the thousand interns win enormously; when you cannot, no GPU will save you.
SIMT: one instruction stream, thousands of threads
How does a GPU organize a thousand workers without a thousand separate control units? The answer is a clever middle path called SIMT — Single Instruction, Multiple Threads. You, the programmer, write code as if for one thread: 'take my element, square it, add the result'. You then launch that one small program — a kernel — across thousands of threads at once, each handed its own data index. It feels like a thousand tiny independent programs.
Underneath, the hardware does something thriftier. It bundles threads into fixed-size groups — NVIDIA calls a group of 32 a warp — and the whole warp marches in lockstep, all 32 threads executing the same instruction in the same cycle, just on their own 32 different data values. So one instruction-fetch and one decode are shared across 32 lanes of arithmetic. That is the SIMT bargain: it looks like independent threads to you (easy to program), but runs like wide SIMD underneath (cheap to build). It is genuinely a blend of both worlds, and naming it lets us be precise about where it bites.
And here is exactly where it bites: branches. Because a whole warp shares one program counter, what happens when its 32 threads hit an `if` and some want to take the branch while others do not? The hardware cannot run two different instructions in one cycle, so it does something honest but costly — it runs the `if` side with the 'else' threads switched off, then runs the `else` side with the 'if' threads switched off, executing both paths in sequence. That serialization is called branch divergence, and it is the single most important performance pitfall of the GPU.
Warp of 8 threads hits: if (x > 0) A(); else B(); thread: t0 t1 t2 t3 t4 t5 t6 t7 x>0? Y N Y Y N N Y N cycle 1 run A() : [t0 -- t2 t3 -- -- t6 --] (N threads idle) cycle 2 run B() : [-- t1 -- -- t4 t5 -- t7] (Y threads idle) Both halves ran one after the other. Threads that ALL agree on the branch cost nothing extra; a split warp pays for BOTH paths -> up to 2x slower here.
Hiding latency by drowning it in work
There is a beautiful idea hiding in how a GPU stays busy, and it is the opposite of the CPU's strategy. A CPU hates memory latency and spends huge effort avoiding it: deep caches, prefetchers, out-of-order execution to find other work during a cache miss. A GPU mostly does not bother trying to avoid latency at all — instead it hides it by having so many warps ready to run that there is always another one to switch to.
- A warp issues a load from slow DRAM and will need to wait hundreds of cycles for the data to arrive.
- Instead of stalling, the scheduler instantly parks that warp and switches to another warp that already has its data ready — and runs it.
- When that one also hits a load, switch again — and again — cycling through dozens of warps, the arithmetic units never sitting idle.
- By the time the scheduler comes back around, the first warp's load has long since completed, so it resumes with no wait. The latency was real, but it was buried under other work.
This is why a GPU needs thousands of threads in flight, far more than it has lanes: the surplus is not for raw parallelism but as a deep reservoir of work to hide every stall. It is also why the strategy fails the instant you do not have enough independent work — with only one warp, that load stalls the whole machine and the GPU just sits there, hundreds of fast lanes frozen, waiting like the slowest possible CPU. Throughput hiding only works when the work keeps flowing.
Roofline: when are you even using the machine?
Suppose your problem really is GPU-shaped and you run it. How do you know whether the chip is actually being used well, or starving? There is one elegant picture that answers this for both CPUs and GPUs: the roofline model. It rests on a single ratio called arithmetic intensity — the number of useful arithmetic operations you perform for each byte of data you fetch from memory. High intensity means lots of math per byte (a small piece of data reused many times); low intensity means you barely compute before running back to memory for more.
Plot achievable performance against that intensity and you get a roof with two parts. On the left, at low intensity, you are memory-bound: a sloped ceiling rising with intensity, because your speed is capped by how fast memory bandwidth can deliver bytes — the lanes are mostly waiting. On the right, at high intensity, you are compute-bound: a flat ceiling at the chip's peak arithmetic rate, because now you have enough math per byte to keep every lane busy. Every workload sits somewhere under this roof, and the model tells you which wall you are hitting.
The lesson is bracing and honest: a GPU's giant peak number is only reachable if your arithmetic intensity is high enough to stand under the flat part of the roof. A great many real programs are stuck on the sloped, memory-bound side, so they touch only a sliver of the advertised speed no matter how many lanes the chip has. This is also why moving data between CPU and GPU over the PCIe link can quietly dominate your run time — if you ship a big array across just to do a little math on it, your intensity is dismal and the transfer, not the computation, is the real cost. The roofline keeps you honest about whether you are using the machine or merely owning it.
The honest verdict: a throughput machine, not a faster CPU
Pull it together and a clean contrast emerges. The CPU is a latency machine: it pours its transistors into finishing one dependent chain of work as soon as possible, and it is superb at serial, branchy, decision-heavy code. The GPU is a throughput machine: it pours its transistors into doing a colossal amount of independent work per second, and it is superb only when that work is abundant, uniform, and arithmetic-heavy. They are not points on one 'fast' line; they are tuned for opposite questions.
So here is the truth a marketing slide will never put plainly: a GPU is genuinely bad at exactly the things a CPU is good at. Hand it serial code where step two needs step one's answer, and its thousands of lanes sit idle while one limps along. Hand it branchy code full of unpredictable `if`s, and warps diverge until you crawl. Hand it latency-sensitive work — respond to this one request now — and it has nothing clever to make a single thread finish fast. A GPU is not a faster CPU; it is a different shape of machine that happens to be wired into the same box.
That contrast is exactly why modern systems are heterogeneous, pairing a CPU and a GPU and letting each do what it is shaped for. The CPU runs the program's serial spine and its messy decisions; it hands the GPU the fat, uniform, parallel loops to devour. The next two guides sharpen this: guide 4 dissects SIMT, warps, and divergence in mechanical detail, and guide 5 builds a practical checklist for choosing which machine a given piece of work belongs on. Hold the one-line summary until then: the GPU trades brilliance at one task for relentlessness at a million.