the memory hierarchy

Imagine a cook with a tiny cutting board, a small counter, a pantry across the room, and a warehouse across town. The cutting board is instant to reach but holds almost nothing; the warehouse holds everything but every trip there is slow. A good cook keeps whatever they are using right now on the board and the counter, and only walks to the warehouse rarely. Your CPU faces exactly this trade-off, and the memory hierarchy is its answer: several layers of storage, each one bigger and slower than the one above it.

From fastest and smallest to slowest and largest, the layers are: the registers (a few hundred bytes, read in well under one nanosecond, essentially free), the L1 cache (typically 32 to 64 KiB per core, around 4 cycles or roughly 1 ns), the L2 cache (256 KiB to a few MiB, around a dozen cycles), the L3 or last-level cache (several to tens of MiB, shared between cores, tens of cycles), and finally main memory or DRAM (gigabytes, but 60 to 100+ nanoseconds away — well over a hundred cycles). Each step down is roughly an order of magnitude slower. The CPU does not let you pick a level: it automatically keeps recently and nearby-used data in the fast caches and fetches from DRAM only on a miss.

Why this matters to your code: the gap between an L1 hit and a DRAM access is the single largest performance cliff most programs ever fall off. A processor that issues several instructions per nanosecond can stall for the equivalent of hundreds of instructions waiting on one DRAM read. This is why locality of reference wins — code that touches memory that is already in cache, or close to data it just touched, can run many times faster than identical-looking code with a scattered access pattern. The numbers above are typical, not exact; they vary by chip, but the shape — a steep latency gradient — is universal.