From a guess to a gamble: speculative execution
Guide 4 ended with the branch predictor making an educated guess about which way an if will go, so the pipeline need not stall while the condition is still being computed. Speculative execution is what the core does with that guess: it does not merely fetch the predicted path, it actually runs it — performing the additions, the loads, the comparisons of instructions it is not yet certain it should be executing at all. The results pile up in the reorder buffer in a kind of pending, not-yet-real state. Only when the branch finally resolves does the core decide whether to commit that work (make it permanently visible in the registers and memory) or squash it and pretend it never happened.
Picture a tiny loop that walks an array and sums only the positive entries. The branch on `if (a[i] > 0)` is taken or not taken depending on data the core cannot see in advance. With prediction plus speculation, the core does not wait: it predicts "probably positive", speculatively executes the add, and keeps the pipeline full. If the prediction was right, the add was already done — pure profit. If it was wrong, the out-of-order machinery rolls the architectural state back to the branch and re-steers down the correct path. This is why a well-predicted branch is nearly free and a badly predicted one costs a dozen-plus cycles: the misprediction does not just stall the pipe, it throws away real work the core already performed.
SIMD: one instruction, many lanes
Everything so far has chased more instructions per clock. SIMD chases more data per instruction. The name stands for Single Instruction, Multiple Data: instead of an add register that holds one 32-bit int, the core has wide vector registers — 128, 256, even 512 bits — and a single add instruction that operates on all the lanes at once. A 256-bit register holds eight 32-bit floats; one vector add adds all eight pairs in the time a scalar add handled one. The lanes are independent: lane 3 never talks to lane 4. It is the assembly line made literal — eight identical workers doing the same step on eight items in lockstep.
Scalar: one 32-bit float per instruction add s0, s1, s2 ; s0 = s1 + s2 (1 result) SIMD: eight 32-bit floats in a 256-bit register, one instruction [ a0 a1 a2 a3 a4 a5 a6 a7 ] <- vector register v1 + [ b0 b1 b2 b3 b4 b5 b6 b7 ] <- vector register v2 --------------------------- [ c0 c1 c2 c3 c4 c5 c6 c7 ] <- one vector add, 8 results Lanes are independent: c3 = a3 + b3, and never touches lane 4.
You rarely write these instructions by hand. The compiler's job is auto-vectorization: at higher optimization levels (think gcc -O2 or -O3) it tries to recognize a loop whose iterations are independent and rewrite it to process several elements per pass with vector instructions. The honest caveats matter. Auto-vectorization is fragile: a loop-carried dependency (where iteration i needs iteration i-1's result), a function call inside the loop, unpredictable branches, or pointers the compiler cannot prove are non-overlapping will all silently disable it. There is no error, no warning by default — your loop just stays scalar and slow. This is exactly why vectorization rewards the mechanical sympathy this rung is teaching: code written with the hardware's grain runs many times faster than code written against it, even though both are "correct".
Prefetching: fetching memory before you ask
Guide 1 of this rung set up the brutal fact behind everything else: a cache miss that goes all the way to main memory costs on the order of a hundred-plus cycles, during which even an out-of-order core eventually runs out of independent work and stalls. Prefetching is the hardware's defense. The hardware prefetcher is a small unit that watches your stream of memory accesses, detects a pattern — most commonly a constant stride, like "every access is 64 bytes past the last" — and speculatively issues loads for cache lines it predicts you will touch next, so the data is already sitting in cache by the time the instruction that needs it arrives. Done well, the hundred-cycle miss simply never happens.
The dependency on pattern is the whole story, and it is what ties this guide back to the start of the rung. Walking an array straight through (stride of one element) is the prefetcher's dream — it locks on instantly. Chasing a linked list, where each node's address is whatever the last node happened to store, is its nightmare: the next address is unknowable until the current load completes, so there is no stride to detect and every node is a fresh cache miss. This is the concrete, cycle-counting reason a flat array of values often crushes a "more elegant" pointer-linked structure holding the same data. The algorithm's big-O can be identical while one runs five times faster, purely because one feeds the prefetcher and the other starves it.
Three engines, one budget
Step back and notice these three tricks attack three different bottlenecks. Speculation and the pipeline fight control hazards — the cost of not knowing which instruction comes next. SIMD fights arithmetic throughput — the cost of doing one operation at a time when the work is wide and uniform. Prefetching fights memory latency — the cost of waiting on data that is far away. A real core runs all three at once, layered: a speculatively-executed, predicted-taken loop body, full of vector instructions, marching through an array that the prefetcher is feeding from a few cache lines ahead. That stacked picture is what "a modern CPU is fast" actually means.
And every one of them is speculative or pattern-bound, which is the honest theme of the whole rung. The branch predictor guesses; speculation runs work that might be thrown away; the prefetcher bets on a stride; auto-vectorization assumes your iterations are independent. When your code matches what the hardware expects — predictable branches, independent loop bodies, sequential memory — these engines run at full tilt and the machine feels magical. When it does not — a data-dependent branch the predictor cannot learn, a loop-carried dependency, a pointer chase — the engines stall, mispredict, and starve, and the same algorithm runs many times slower. You do not control these units directly; you control whether your code gives them something they can win on.
- Make branches predictable, or remove them. A branch that goes the same way nearly every time is essentially free; a 50/50 data-dependent one is a misprediction machine. When you cannot make it predictable, see whether branchless code (a conditional move, a mask, an arithmetic trick) is faster — but measure, because the branch predictor is often smarter than you expect.
- Write loops the compiler can vectorize: independent iterations, no surprise function calls inside, no loop-carried dependency, and use restrict (or local copies) so the compiler can prove pointers do not overlap. Then check the optimizer's report (gcc -fopt-info-vec) to confirm it actually vectorized — do not assume.
- Lay data out for sequential access so the hardware prefetcher locks on: prefer a flat array to a pointer-linked structure for hot data, keep the fields you touch together adjacent, and walk memory in a straight stride. This single habit often beats every micro-optimization, because it turns hundred-cycle misses into hits.
- Measure, never guess. All three engines are invisible from the source code, so the only honest way to know whether they are working is a profiler with hardware performance counters (mispredicted branches, cache misses, instructions per cycle). The slogan for this whole rung is mechanical sympathy: understand the machine well enough to write code it can run fast — then prove it with numbers.