Two machines, two goals
By now the two chips on your desk should feel like different species, not different speeds. A CPU is built to finish one chain of work as fast as possible — to minimize response time. That is why it spends transistors on out-of-order execution, deep branch prediction, and big caches: all machinery for racing a single, twisty, dependency-laden instruction stream to the finish line. A GPU makes the opposite bet. It barely cares how long any one item takes; it cares about total work per second — throughput — so it pours its silicon into thousands of simple lanes and hides latency by always having another warp ready to run.
Hold the old analogy clearly in mind: the CPU is one brilliant chef who can improvise around any obstacle in the recipe, while the GPU is a thousand interns each doing one tiny sum. Ask the chef to run a marathon of identical additions and the interns crush them; ask the interns to follow a winding recipe full of 'if the sauce splits, do this instead', and they fall apart while the chef shrugs and adapts. Neither is better — they answer different questions. The whole skill of this guide is learning to recognize which question your code is asking.
The roofline: is your job starved or busy?
To decide where a job belongs, we need one number about the job itself, not the chip. That number is arithmetic intensity: how much math you do per byte you move from memory. Define it as floating-point operations divided by bytes touched. A loop like `c[i] = a[i] + b[i]` reads 8 bytes and writes 4 (single precision) to do one add — intensity near 0.1 flops/byte, almost pure data movement. Multiplying two large matrices reuses each loaded value across a whole row of work — intensity in the tens or hundreds. Same hardware, wildly different demands on the memory system.
The roofline model turns this into a single picture. Plot achievable performance (flops/second) against arithmetic intensity. There are two ceilings. A sloped roof on the left is the bandwidth limit: if you do little math per byte, your speed is capped by how fast memory can feed you — peak bandwidth times intensity. A flat roof on the right is the compute limit: once intensity is high enough, you are bounded by the chip's peak arithmetic, not its memory. Where the two roofs meet is the ridge point. Your kernel sits somewhere under these roofs, and which roof it hits tells you what to fix.
performance
(flops/s)
^
peak|............______________________ <- compute roof (flat)
| /:
| / : matmul lives here
| / : (compute-bound)
| / : *
| / :
| / <-- bandwidth roof (slope = peak bandwidth)
| /:
| / : * array-add lives here
| / : (bandwidth-bound)
+-/---+---------------------------> arithmetic intensity
ridge (flops per byte)
Left of the ridge: memory-bound -> raise intensity / reuse data.
Right of the ridge: compute-bound -> the GPU's thousands of lanes shine.When the GPU wins big
The GPU shines when the work is embarrassingly parallel: thousands of independent items, the same operation on each, and enough arithmetic per byte to keep the lanes busy rather than starved. This is exactly the data-level parallelism of the earlier guides, scaled up. Dense linear algebra, image and video filters, physics on a grid, and the multiply-add storm at the heart of neural networks all sit comfortably to the right of the roofline ridge, where the GPU's thousands of lanes are the whole point. Feed such a kernel enough threads and the hardware hides memory latency by swapping in another warp whenever one stalls — latency hidden, throughput maximized.
But raw arithmetic is only half the win; the other half is bandwidth. A GPU pairs its many lanes with very wide, very fast memory — think high-bandwidth memory stacked beside the chip — precisely because thousands of lanes would otherwise starve. This is the same lesson SIMD taught, now at planet scale: the adders are easy, feeding them is the engineering. A GPU's advantage on a bandwidth-bound kernel is often less about flops and more about that fat pipe to memory. Read it off the roofline: a steeper bandwidth roof, not a higher compute roof, is what carries the bandwidth-bound code.
When the GPU is the wrong tool
Now the honest half. A GPU is genuinely poor at serial, branchy, latency-sensitive code, and pretending otherwise wastes real money. Serial: if step N depends on step N-1, you cannot spread the work across thousands of lanes — the chain has length, not width, and 999 of your 1000 interns stand idle while one works. The CPU, with its aggressive out-of-order engine, is built precisely to race a single dependency chain. Branchy: recall branch divergence from the SIMT guide — when threads in a warp take different paths, the hardware runs both paths and masks off the lanes that should not have run. Code full of unpredictable `if`s drags a GPU toward a fraction of its peak.
Latency-sensitive: the GPU hides latency by having lots of other work. If a task is small or must respond right now — handle one keystroke, run a short query, react within a millisecond — there is no parade of warps to swap in, so the GPU's whole strategy collapses and you are left with a slow, low-clock core. And there is a tax you pay before the GPU runs at all: host-device transfer. The data must travel from CPU memory across a PCIe link into GPU memory, and the results back. If the kernel is quick, this copy can cost more than the computation you offloaded — a classic way to make code slower by 'accelerating' it.
The real answer: use both
The title asks you to pick a tool, but the mature answer is rarely 'one or the other'. Real systems are heterogeneous: a CPU runs the serial spine of the program — the branchy control logic, the part-by-part decisions, the parts that must respond now — and hands the big, regular, parallel slabs of math to the GPU. The CPU is the host, orchestrating; the GPU is the device, grinding through the throughput work. This is the same division of labour that has driven the whole post-Dennard story: when the clock stopped getting faster around 2005, the industry turned to many cores and then to specialized engines, each doing what it does best.
Carry one mental model away. Match the shape of the work to the shape of the machine: a single twisty dependency chain wants the latency-hunting CPU; a wide ocean of identical, independent, arithmetic-heavy work wants the throughput GPU. The roofline tells you whether you are starved or busy; arithmetic intensity and transfer cost tell you whether the trip across PCIe is worth it. Picking the right tool is not loyalty to a chip — it is honestly reading the job, and most ambitious programs end up needing both tools working together.