The polite lie of one-at-a-time
From the assembly rung you carry a clean mental model of the instruction cycle: the CPU reads the instruction at the address in its program counter, decodes it, does whatever it says, updates registers and flags, advances the program counter, and repeats. One instruction fully finishes before the next one starts. It is a beautiful model and you should keep it — for reasoning about what a program means, it is exactly right. The machine behaves as if this is what happens. But it is not how a real CPU spends its time, and the gap between the model and the silicon is where this whole rung lives.
Here is the pressure that broke the simple model. A modern core runs at, say, 3 GHz — one cycle is about a third of a nanosecond. From the earlier guides you already know that an L1 cache hit costs a handful of cycles, an L3 hit a few dozen, and a trip to DRAM well over a hundred. If the CPU truly did one instruction start-to-finish before touching the next, then every single instruction would run only as fast as its slowest stage, and the fast stages would sit idle waiting. That is a catastrophic waste of silicon. The fix is the same trick a kitchen uses at the dinner rush: you do not cook one meal completely before starting the next — you overlap them.
The pipeline: an assembly line for instructions
Break the work of one instruction into stages — classically five: fetch the instruction bytes, decode what they mean, execute the arithmetic, access memory for any load or store, and write back the result into a register. Now build a separate piece of hardware for each stage, lined up in a row, and let an instruction march through them one stage per cycle. The trick is that while instruction 1 is in execute, instruction 2 can be in decode, and instruction 3 in fetch — three instructions in flight at once, each in a different stage. This is the instruction pipeline, and it is the single most important idea in how a CPU goes fast.
cycle: 1 2 3 4 5 6 7
--- --- --- --- --- --- ---
instr A F D E M W
instr B F D E M W
instr C F D E M W
instr D F D E M W
F=fetch D=decode E=execute M=memory W=writeback
one instruction still takes 5 cycles end to end,
but a *new* one completes every cycle once fullLook carefully at the table and a subtle, crucial distinction jumps out. Each individual instruction still takes five cycles to travel from fetch to write-back — that is its latency, and the pipeline does not shrink it. What the pipeline buys you is throughput: once the pipe is full, a new instruction completes every single cycle. A factory line does not make one car faster, but it rolls a finished car off the end far more often. Real cores stretch this much further — modern x86-64 chips have pipelines fifteen to twenty stages deep — but the principle is exactly this five-stage sketch, just longer.
Where the pipe stalls: hazards
The pretty picture assumed every instruction is independent of its neighbours. Real code is not. Suppose instruction B needs the value instruction A is computing — `a = x + y;` then `b = a * 2;`. B's execute stage wants `a`, but in the table A does not write back `a` until cycle 5, while B reaches execute in cycle 4. B would read a stale register. This is a data hazard: a dependency that the overlap would violate. The blunt fix is to make B wait — insert stall cycles (called bubbles) until A's result is ready. Stalls are pure waste: cycles where part of the pipe does nothing. Hardware fights back with forwarding, wiring A's freshly computed result straight to B's input without waiting for the write-back, but dependencies you cannot forward around still cost real cycles.
The nastier hazard is the control hazard, and it is the villain of this whole guide. When the CPU fetches a conditional branch — the machine code for an `if` or a loop test — it does not yet know whether the branch is taken. The branch's condition often depends on a value that is still being computed deep in the pipe. But the fetch stage cannot wait: it needs to know right now which instruction to fetch next, the one after the branch or the one at the branch target. In a 20-stage pipeline the answer might not be known for a dozen cycles. Stalling the whole front end for a dozen cycles on every branch — and branches are roughly one instruction in five — would throw away most of the throughput the pipeline was built to gain.
Branch prediction: betting on the future
The CPU's answer to the control hazard is audacious: it guesses. Rather than stall, the branch predictor bets on which way the branch will go, and the front end charges ahead fetching and executing instructions down the predicted path — speculatively, before anyone knows if the guess was right. If the guess holds, those instructions were exactly the work that needed doing, and not a single cycle was lost: the branch became free. The predictor is not a random coin flip; it is a small, fast learning machine that watches the history of each branch. The key insight that makes it work is that branches are wildly predictable: a loop's back-edge is taken hundreds of times then not-taken once; an error-check `if` is almost never taken. Modern predictors exceed 95% accuracy on typical code, often 99%.
But a guess can be wrong, and that is where the bill comes due. On a misprediction the CPU has spent a dozen cycles fetching and partly executing instructions from the wrong path. None of their results may be allowed to stick — they were never supposed to run. The hardware must flush them: throw away all that speculative work, rewind the front end, and restart fetching from the correct address. That flush-and-refill is the branch misprediction penalty, and on a deep pipeline it is real money — commonly fifteen to twenty cycles, a hole as expensive as a trip to L3 cache. So a branch is not uniformly cheap or costly: it is nearly free when predicted right and surprisingly painful when predicted wrong, and the difference is determined by a pattern the predictor either has or has not learned.
Superscalar and out-of-order: many at once, finished in order
A pipeline completes at most one instruction per cycle. The next leap is to widen it: build several execute units side by side, fetch and decode a handful of instructions together, and dispatch independent ones to run in parallel in the same cycle. This is superscalar execution, and it is how a chip exploits instruction-level parallelism — genuinely retiring two, three, four instructions in a single cycle. The catch returns immediately: instructions can only run together if they do not depend on each other. A long chain where each instruction needs the previous one's result — like a pointer chasing down a linked list — exposes almost no parallelism, and all that extra execute hardware sits idle no matter how wide the chip is.
Now the boldest idea of all. If instruction B is stalled waiting on a slow load from DRAM, but instruction C right behind it is independent and ready, why should C wait politely in line? It should not. An out-of-order core lets C run while B is still stuck, executing instructions in whatever order their inputs become available rather than the order the program wrote them. The core keeps a window of dozens — sometimes hundreds — of in-flight instructions, plucking ready ones out to execute the moment their operands arrive. This is how a CPU hides a cache miss: instead of stalling for 200 cycles waiting on DRAM, it finds other independent work to do during the wait, keeping the execute units busy.
But this raises a frightening question: if instructions finish in a scrambled order, how does the program stay correct? If C finishes before B and B then faults — say it divides by zero or hits a bad pointer — the architecture must behave as if C never ran. The answer is the reorder buffer. Instructions may execute out of order, but they retire — commit their results to the visible architectural state — strictly in program order. The reorder buffer is a queue that holds each instruction's completed-but-not-yet-committed result until every older instruction ahead of it has safely retired. This is the deep reconciliation of the whole guide: chaos in execution, perfect order at retirement, so the simple one-at-a-time model you started with remains exactly true from the outside.
Why this matters at your keyboard
All three engines — pipeline, branch predictor, out-of-order window — share one purpose: keep the execute units busy every cycle, never stalling. And they share one Achilles' heel: they only help when there is independent, predictable work to find. A tight loop over an array, with a predictable exit test and no long dependency chains, lets the core stretch its legs and run near its peak instruction-level parallelism (ILP). A pointer chase through a linked list, or a branch that flips unpredictably, starves all three at once — and the same chip that ran the array loop at four instructions per cycle now crawls at a fraction of one. This is the honest reason two programs doing the same amount of arithmetic can differ tenfold in speed.
There is one more thread to pull, and it leads straight into the next guide. Notice that the moment the branch predictor guesses, the out-of-order core does not just fetch down the predicted path — it runs those instructions, computing real results, all before anyone has confirmed the guess. That is speculative execution, and on a misprediction the reorder buffer dutifully throws those results away so the program stays correct. For decades that looked like a perfectly safe optimisation. It was not quite: the speculated work leaves faint footprints in the caches you studied earlier, and those footprints can be read out even after the results are discarded. That crack is where Spectre and Meltdown live — and unpacking it is exactly where guide 5 begins.