the memory wall
Imagine a chef who, over the years, learns to chop ten times faster — but the pantry is still the same long walk down the hall. Soon the chef spends almost the whole shift walking to the pantry, not cooking. Speeding up the chef barely helps anymore; the trip to the pantry is now what limits dinner. The memory wall is this same predicament for processors: CPUs got dramatically faster decade after decade, while main memory got faster only slowly, so fetching data became the dominant bottleneck.
Put as a number: from roughly the 1980s onward, processor speed improved by around 50 percent per year for a long stretch, while DRAM access time improved only single-digit percent per year. The two curves diverge exponentially, and the gap is the 'wall'. A modern core can execute hundreds of instructions in the time it takes one access to main memory to return. So if the processor had to wait for memory on every access, it would sit idle the vast majority of the time, and all its speed would be wasted.
The memory wall is the reason caches exist and the reason cache design is arguably the most important performance topic in computer architecture. Caches, prefetching, deeper hierarchies, and wider memory channels are all weapons against the wall. It is worth being honest: the wall has not been demolished, only managed — memory latency is still hundreds of times worse than register access, and a single cache miss to DRAM can stall a fast core for the equivalent of hundreds of instructions. Much of the cleverness in modern chips and in cache-friendly code is, at bottom, about not hitting the wall.
If a core runs at 3 GHz (one cycle ~0.33 ns) and a main-memory access takes ~80 ns, that one access is about 240 cycles. During those cycles a wide core could have retired hundreds of instructions — all lost to waiting, unless a cache catches the access first.
The CPU outran memory by orders of magnitude; the growing gap is the wall caches were built to hide.
The wall is about LATENCY (how long one access takes), which improves very slowly. Memory BANDWIDTH (bytes per second) has grown much faster — so techniques that turn latency-bound access into bandwidth-bound streaming (prefetching, good locality) help a lot.