What we built, and the question it forces
Stand back and look at the engine these four guides assembled. A superscalar front end fetches and issues several instructions per cycle. Out-of-order execution lets any instruction whose operands are ready run early, while the reorder buffer still retires them in program order so the visible result stays sequential. Register renaming dissolves the false dependences that would otherwise serialise everything, and speculative execution guided by a good branch predictor runs past branches before we know they were taken. Every trick has one goal: find more instruction-level parallelism — independent work to do in the same cycle.
So here is the honest question this rung has been building toward: if we keep widening the machine — eight-wide, sixteen-wide, an ever bigger instruction window — does performance keep climbing? It does not, and not by a little. The available parallelism inside one ordinary program is finite, and well before we run out of transistors we run out of independent instructions to feed them. This ceiling has a name: the limits of ILP.
Why the parallelism runs out
Three walls close in at once. First, true data dependences: if instruction B genuinely needs the value A computes, no amount of cleverness lets B run before A — renaming removes the false dependences but never the real ones, and a long chain of dependent operations is a rope the machine must climb one knot at a time. Second, branches: real code branches every five or six instructions, and to see far enough ahead to find independent work the processor must predict many branches correctly in a row. Even a 95%-accurate predictor, raised to the tenth power, is right end-to-end only about 60% of the time — so the useful window of speculation is shorter than it looks.
Third, and most decisive, cost grows faster than benefit. To exploit more parallelism you need a bigger reorder buffer, more reservation stations, more rename registers, more read/write ports on the register file, and wider wake-up logic — and several of those structures grow roughly with the square of the issue width. You spend a lot more silicon and a lot more energy to coax out a little more speed. This is the power and complexity cost of ILP: the classic studies found that beyond about four-wide issue, real programs yield sharply diminishing returns. A wider machine that is idle most of the time is just an expensive heater.
Two doors out: more threads, then more cores
If one stream cannot keep a wide machine busy, why not feed it from two streams? That is simultaneous multithreading — SMT, or Intel's Hyper-Threading. The core already has more execution slots than a single thread can usually fill, so SMT lets two (or more) threads issue into the same back end in the same cycle. When one thread stalls on a cache miss, the other keeps the units warm. It is a thrifty trick — almost free silicon, since it reuses the engine you already built — but it shares, not multiplies, the resources, so two threads rarely run twice as fast, and they can even slow each other by fighting over the cache.
The bigger door is the one the industry actually walked through around 2005: stop making one core ever wider and put several whole cores on the chip — the multicore processor. This was not a fashion; it was forced. Dennard scaling — the happy rule that shrinking transistors let each generation run faster at the same power — effectively ended around 2005, so cranking the clock or the issue width any further melted the chip. With single-stream ILP tapped out and power capped, the only way left to spend the still-growing transistor budget was duplication: more cores, exposing thread-level parallelism instead of squeezing harder on one thread.
The honest scar: Spectre and the cost of speculation
Now the uncomfortable part. We told ourselves a reassuring story in guide four: speculation is safe because when a branch is mispredicted the machine squashes the wrong-path instructions — it throws away their registers, never commits them, and rolls the architectural state back as if they never ran. The visible result is correct. But "never ran" turned out to be a half-truth. The wrong-path instructions did not change any register you can name — yet while they were briefly alive, they could touch the cache, and a touched cache line stays warmer than an untouched one. They left a footprint in the microarchitectural state that rollback does not erase.
That footprint is a side channel. The result of a squashed instruction was thrown away, but the time it takes to read memory afterward silently encodes what that instruction touched: a cache hit is fast, a miss is slow, and you can tell them apart with a stopwatch. Spectre weaponises exactly this. An attacker trains the branch predictor to mispredict, tricks the core into speculatively reading a secret it is not allowed to use, and lets that secret steer which cache line gets loaded. The architectural state is dutifully rolled back — but the attacker then times a cache-timing probe to read out which line was warmed, recovering the secret one bit at a time, straight through the protection boundary the hardware swore to enforce.
1. Train the predictor so a bounds check is predicted 'in range'.
2. Speculatively run: secret = array[evil_index]; // out of bounds!
3. Speculatively run: junk = probe[secret * 64]; // touches one cache line
4. CPU discovers the misprediction -> squashes 2 and 3, rolls registers back.
5. BUT probe[secret*64] is now cached. Time each probe[i]:
the one fast access reveals 'secret'. One byte leaked.What Spectre really teaches
Spectre's sibling Meltdown is narrower — it exploited one vendor's habit of letting a load see kernel data speculatively before the permission check finished, and a microcode and operating-system fix largely closed it. Spectre is deeper and harder, because it abuses correct, by-the-book speculation that every high-performance core relies on. There is no single clean patch; defences are a patchwork of compiler fences, predictor flushing on context switch, and disabling some speculation in sensitive code — each of which gives back a slice of the very performance speculation was added to buy. It is the cleanest possible proof that performance and security can be in genuine tension.
- ILP is finite: true data dependences and branches cap how much one stream can parallelise, and cost grows roughly with the square of issue width.
- Around 2005 Dennard scaling ended, so the industry stopped widening one core and turned to SMT and then multicore.
- Multicore shifts the work of finding parallelism onto the programmer, and Amdahl's law caps the payoff by the serial fraction.
- Speculation leaves microarchitectural traces; Spectre reads those traces through a cache-timing side channel, leaking secrets across a protection boundary.
Step back and see the whole arc of this rung. We discovered enormous hidden parallelism inside an ordinary instruction stream and built astonishing machinery — out-of-order issue, renaming, Tomasulo's algorithm, the reorder buffer, branch prediction — to harvest it while keeping results perfectly in order. Then we met the wall: that parallelism is finite and that machinery is expensive, which is exactly why the next rungs of this ladder turn to many cores, GPUs, and accelerators. And we end on an honest note: the cleverness that won us decades of speed is the same cleverness Spectre turned against us. Great architecture is not just making it fast — it is understanding, and owning, what fast really costs.