The free lunch nobody noticed they were eating
Look back down the ladder at everything we built to make one core fast. We deepened the pipeline so several instructions are in flight at once, went superscalar to issue more than one per cycle, added out-of-order execution and branch prediction to keep the machine busy across stalls, and wrapped the whole thing in a deep memory hierarchy so the data is usually waiting on the desk. For about thirty years, this kept paying off in a way that felt like magic to programmers: write ordinary single-threaded code, do nothing special, and next year's chip simply ran it faster.
That gift had two engines. Moore's law kept doubling the number of transistors on a chip every couple of years, giving architects more raw material for bigger caches and wider pipelines. And Dennard scaling — the quieter, more important one — said that as transistors shrank, their voltage and current dropped in step, so a smaller transistor used proportionally less power and could switch faster. Together they let designers crank the clock rate up generation after generation without the chip melting. The free lunch was really Dennard scaling paying the power bill.
The power wall: why the clock stopped climbing
Around 2005 the bill came due. Dennard scaling effectively ended: transistors kept shrinking, but their voltage could no longer drop in step, because below a certain voltage they leak current even when they are supposed to be off. So packing more, faster-switching transistors into the same area no longer kept power flat — it drove power up. The rough rule is brutal: dynamic power grows roughly with frequency times voltage-squared, so chasing a higher clock costs power super-linearly. Push the clock past about 4 GHz and you are trying to dissipate the heat of a kitchen hotplate from a fingernail of silicon. That hard limit is the power wall.
Here was the architects' dilemma in 2005. Moore's law was still working — every two years they were handed roughly twice as many transistors. But they could no longer spend those transistors the old way, on a single faster core, because the power wall forbade a faster clock and the limits of instruction-level parallelism meant a wider single core gave diminishing returns at a steep power cost. So they made a different bet: instead of one ever-bigger, ever-hotter core, place two or more complete cores on the same chip. The multicore processor was born not out of triumph but out of necessity — it was the only way left to turn extra transistors into extra performance.
What multicore gives, and what it asks in return
Two cores can do twice the work per second — but only if there are two independent streams of work to do. That is the catch. A multicore chip exploits thread-level parallelism: it runs several independent sequences of instructions, called threads, genuinely at the same time, one per core. Where a single core squeezed parallelism out of one instruction stream (the pipelining and out-of-order tricks of earlier rungs), multicore parallelism comes from multiple streams running together. The hardware has handed the programmer a new job: find the independent work and hand it to the cores as threads.
The most common way to organize this is a shared-memory multiprocessor: every core sees the same single address space, so threads communicate just by reading and writing the same memory locations. When the cores are identical and all reach memory through the same path, we call it symmetric multiprocessing (SMP). This is wonderfully convenient — one thread writes a result, another simply reads it, no explicit hand-off needed. The alternative, message passing, gives each core its own private memory and makes them mail data to one another; that scales to huge clusters but is more work to program. For the cores on a single chip, shared memory is the usual choice.
Shared memory is convenient, but it quietly creates the central problem of this whole rung. Each core has its own private cache — its own desk. So if two cores both load the same variable, each ends up with its own photocopy of that memory line on its own desk. Now one core writes a new value to its copy. The other core's copy is suddenly stale, and nothing has told it. This is exactly the picture of several people editing photocopies of one page and needing to agree on what the page says — the cache coherence problem — and it is the subject of the very next guide.
A map of parallel machines: Flynn's taxonomy
Before we dive into coherence, it helps to place multicore on a small map of every way a machine can be parallel. The classic one is Flynn's taxonomy, which sorts machines along two axes: how many instruction streams are running, and how many data streams they act on. Just two yes/no questions give four boxes, and almost every real machine lands in one of them.
one data stream many data streams
+----------------------+----------------------+
one instruction | SISD | SIMD |
stream | a classic single | one instruction, |
| core; the von | many data lanes: |
| Neumann machine | GPUs, vector units |
+----------------------+----------------------+
many instruction | MISD | MIMD |
streams | (rare; e.g. some | independent cores, |
| fault-tolerant | each its own work: |
| pipelines) | MULTICORE lives here|
+----------------------+----------------------+Multicore lives in the MIMD box: many independent cores, each with its own instruction stream working on its own data. That is a different kind of parallelism from the SIMD box, where one instruction drives many data lanes at once — the world of vector units and the GPU, a thousand interns each doing one tiny sum in lockstep. A GPU is not just a faster CPU; it is a fundamentally different shape of machine, brilliant at wide regular data but bad at serial, branchy code. This rung is about MIMD multicore; the SIMD and GPU world is a later rung of its own.
The honest ceiling: Amdahl's law
It is tempting to think eight cores means eight times the speed. They almost never do, and there is a precise reason why — one honest enough to deserve its own guide at the end of this rung. Amdahl's law says that the speedup you can get from parallelism is capped by the fraction of your program that is inherently serial and cannot be split across cores. If even a small slice must run one step at a time, that slice becomes the floor your speedup can never get below.
Put numbers on it. Suppose 95% of a program can be parallelized perfectly but 5% is stubbornly serial. With infinitely many cores, the parallel 95% shrinks toward zero time — but the serial 5% never shrinks at all. So the best possible speedup is 1 / 0.05 = 20 times, no matter whether you throw 32, 1000, or a million cores at it. With a realistic 16 cores you would get roughly 1 / (0.05 + 0.95/16), about 9 times, not 16. More cores give sharply diminishing returns, and the serial fraction sets a hard ceiling you simply cannot buy your way past.
One more honest complication sits underneath all of this: not all memory is equally far away. On a big machine with several multicore chips, each chip has its own bank of DRAM attached directly to it. A core reaches its own chip's memory quickly but a neighbour chip's memory more slowly — this is NUMA, non-uniform memory access. It means where your data physically lives now affects speed, and a thread running far from its data pays a quiet tax. NUMA, coherence, consistency, and synchronization are the threads we will pull on through the rest of this rung — all of them consequences of the single decision, forced by the power wall, to stop making one core faster and start making many cores cooperate.