Past one instruction per cycle
In the previous rung you watched a pipeline turn the laundry of a program into an assembly line: while one instruction is washing in execute, the next is in decode, the one after that in fetch. That was a beautiful trick, but notice its quiet ceiling — a simple pipeline still starts exactly one new instruction every cycle. In the best case it retires one per cycle, so its CPI bottoms out at 1. The whole of this rung is about smashing through that floor: building a core whose CPI can go below 1, meaning it genuinely finishes more than one instruction per cycle.
The resource we are mining has a name: instruction-level parallelism, or ILP. The idea is that inside one ordinary, single-threaded program — no threads, no extra cores — there are usually several instructions sitting near each other that do not depend on one another and so could, in principle, run at the very same time. Consider three lines: `a = b + c`, `d = e + f`, `g = a + d`. The first two share nothing; a clever core could compute both adds simultaneously, then do the third. That latent ability to overlap is ILP, and the rest of this rung is the machinery that finds and exploits it.
The superscalar core: more lanes, not a faster clock
The most direct way to exploit ILP is to widen the machine. A superscalar core has several execution units side by side — say two ALUs, a load/store unit, and a multiplier — and fetches, decodes, and issues a small bundle of instructions every cycle instead of one. Picture the five-stage pipeline you know, but two or four lanes wide, like a single-file checkout lane becoming a bank of four registers at the supermarket. If the next few instructions happen to be independent, the core dispatches several into different units in the same cycle and finishes them together.
This is the honest answer to the megahertz myth. People once believed a faster machine simply meant a higher clock rate, but the iron law (run time = instruction count x CPI x cycle time) shows three knobs, not one. A superscalar core does not touch the clock at all; it attacks the CPI term, doing more work per tick. Two cores running at the same gigahertz can differ enormously in real speed because one of them retires four instructions a cycle and the other one. Width, not frequency, is the lever here.
There is a rival design worth meeting now: VLIW, the very long instruction word. A superscalar core figures out at run time, in hardware, which instructions can go together — expensive logic burning power and silicon. VLIW instead pushes that job onto the compiler: it bundles independent operations into one wide instruction ahead of time, and the hardware blindly fires every slot in the bundle in parallel, no scheduling logic needed. The trade is stark and honest: VLIW hardware is simpler and cooler, but the compiler must predict at build time what the hardware would have learned at run time — and it cannot foresee a cache miss or a branch outcome, so its bundles are often padded with wasteful no-ops. This is exactly why mainstream CPUs went superscalar while VLIW thrives mostly in DSPs and some accelerators.
Why order has to bend: dependences
Widening the machine only helps if the next few instructions really are independent — and often they are not. The honest obstacle is dependences. A true dependence (a read-after-write, the RAW you met as a pipeline hazard) is fundamental: if instruction B needs the value instruction A produces, B simply cannot start until A's result exists. No hardware can wish that away. But a second, sneakier kind of obstacle is not fundamental at all.
Consider this trace, where two unrelated computations were both told by the compiler to use register x5:
I1: x5 = x6 + x7 # produces a result in x5 I2: x8 = x5 + x9 # TRUE dependence: really needs I1's x5 I3: x5 = x10 + x11 # reuses the NAME x5 - but new, unrelated work I4: x12 = x5 + x13 # depends on I3's x5, not I1's I3 must wait for I1/I2 only because both grabbed the name 'x5'. The VALUES are independent; only the NAME collides. -> false dependence
I3 reuses the name x5 for brand-new work that has nothing to do with I1. The hardware, seeing the same register number, fears a conflict and serializes them — even though the actual values are independent. This is a false dependence: a stall caused not by data, but by a shortage of register names. The fix, which the next guide develops in full, is register renaming: the core keeps a large pool of hidden physical registers and quietly maps each fresh write of x5 to a different physical slot, so I1's x5 and I3's x5 stop colliding and can run in parallel. The architecture promises only a handful of architectural registers; the microarchitecture secretly provides far more.
Out of order, but plated in order
Even with renaming, a strictly in-order machine stalls whenever the very next instruction is waiting — perhaps on a slow memory load — while perfectly ready instructions sit behind it, idle. The fix is out-of-order execution: the core starts whichever instruction's operands are ready, not whichever comes next in program order. Think of a busy kitchen that begins cooking whichever order has all its ingredients on hand, rather than rigidly first-come-first-served. The machinery that makes this work — reservation stations and Tomasulo's algorithm — is the subject of guide 3; for now just hold the picture of instructions launching as soon as they can.
This separation of execute whenever / commit in order is the load-bearing idea, and it has a familiar flavour: the ISA contract says results appear one after another, while the microarchitecture is free to cheat underneath as long as nobody can tell. In-order commit is what keeps the lie honest — it is the seam where the wild internal reordering gets sewn back into the sequential story the programmer was promised. Guide 2 in this rung is devoted entirely to how that seam holds.
Guessing the future, and sharing the core
There is one more obstacle to deep ILP: branches. About every fifth or sixth instruction is a branch, and the core cannot know which way it goes until late. Waiting would drain the whole deep, wide pipeline. So modern cores speculate: a branch predictor places a bet on which fork the road takes and the machine charges ahead on the guessed path, executing dozens of instructions before the branch even resolves. It is a bet on which way you will turn at a junction you have not reached yet. Guess right (and good predictors are right well over 95% of the time) and you lose nothing; guess wrong and the core must throw away that whole speculative path and restart — which is exactly why the reorder buffer keeps speculative work uncommitted until the bet is settled.
Predictors grew clever over generations: a two-bit predictor gives each branch a little hysteresis so one surprise does not flip its guess; a correlating predictor notices that branches often depend on how recent other branches went; and a tournament predictor runs several schemes at once and learns, per branch, which one to trust — with a branch target buffer caching where a taken branch lands so the core need not even decode it first. Guide 4 in this rung takes these apart in depth. The key honesty: this is all sophisticated guessing, and the recovery cost of a wrong guess is real.
Even a brilliant out-of-order core cannot always find enough independent instructions to fill its wide back end — sometimes the program is just a chain of dependent steps, and execution units sit idle. One elegant fix shares the wasted slots with a second program: simultaneous multithreading (Intel calls it Hyper-Threading) lets two threads issue into the same core in the same cycle, so when one thread stalls on a cache miss the other keeps the units busy. But this is the edge of the rung's honest punchline: there is a hard ceiling to how much ILP any single stream contains. Past a few-wide core, more transistors buy almost nothing — the limits of ILP — and that wall, together with the end of easy frequency scaling, is precisely why the industry turned to multicore. And the very speculation that makes this core fast is also where Spectre-class side channels hide — mis-speculated instructions leave faint footprints in the cache. Guide 5 closes the rung on exactly that twist.