memory wall
Imagine a brilliant chef who can chop and cook a hundred dishes a minute, but the only way ingredients reach the kitchen is one waiter walking back and forth to a pantry down the hall. Soon the chef spends most of the shift standing idle, waiting for the next basket of vegetables. That is the memory wall: over the decades, the speed at which a processor can do arithmetic has grown far faster than the speed at which memory can deliver the data to work on. The compute got cheap; moving the data did not. So for many real workloads, the true bottleneck is not the math, it is feeding the math.
This gap traces straight back to a limit you met lower down the ladder. When Dennard scaling ended around 2006 and clocks stopped rising, chipmakers answered by adding more cores and wider vector and tensor units, so a modern GPU or TPU can demand staggering amounts of data per cycle. Main memory (DRAM) improved too, but mostly in capacity, not in how fast bytes can stream across the narrow wires and connectors between a separate memory chip and the processor. Every trip off-chip costs time and a lot of energy, often far more energy than the calculation itself. The result is that an expensive accelerator can sit largely starved, its arithmetic units waiting on operands.
Two families of ideas attack the wall, and both are about geometry, not faster transistors. The first is bringing memory physically closer and making the path far wider: High-Bandwidth Memory (HBM) is a stack of DRAM dies joined by through-silicon vias (TSVs) and parked right beside the processor on a silicon interposer, replacing a few skinny off-board traces with thousands of short, parallel ones. The second is moving the compute to the data, or near-memory and in-memory compute, where simple operations happen inside or beside the memory so the bytes barely have to travel. Neither makes a transistor faster; both shorten the distance data must move.
off-package DRAM (a few wide-ish DDR channels):
[CPU/GPU] === bus ===========> [ DRAM chip ] far, narrow-ish, power-hungry
HBM stack on a silicon interposer (data travels ~mm, on 1000s of wires):
+--------+ <- DRAM die
+--------+ <- DRAM die } TSV-stacked
+--------+ <- DRAM die
| logic |
[ GPU ]====||==== thousands of short, parallel links
---------- interposer ----------Off-package DRAM forces data down a few long, narrow channels; HBM stacks DRAM with TSVs and sets it on an interposer beside the processor, trading a few skinny wires for thousands of short, parallel ones to widen the pipe.
The memory wall is fundamentally a bandwidth-and-energy-of-data-movement problem, not a capacity one, which is why packaging and integration, not finer lithography, are doing the heavy lifting against it.