the memory hierarchy
Imagine a cook working at a tiny cutting board. Whatever ingredients are right on the board are grabbed instantly; things in arm's reach on the counter take a moment; items in the fridge across the room take a real trip; and anything down in the basement freezer means a long walk. A computer's memory is organised the same way, as a hierarchy of stores that get bigger but slower the further they are from the processor. This staircase — from registers, to cache, to main memory, to disk — is the single most important fact about why fast code is fast.
Concretely the levels are: a few dozen registers right inside the core (sub-nanosecond, the cutting board); L1 cache (a few tens of kilobytes, ~1 nanosecond); L2 cache (hundreds of kilobytes); L3 cache (a few to tens of megabytes, shared by cores, ~10-20 nanoseconds); main memory or DRAM (gigabytes, ~100 nanoseconds, the basement); and disk or SSD (terabytes, microseconds to milliseconds). Each step down is roughly an order of magnitude bigger and an order of magnitude slower. The hardware automatically keeps recently used data in the small fast levels and falls back to the slow ones only when it must, but it cannot read your mind — it bets that data you just touched, and its neighbours, will be needed again soon.
The reason this matters for numerical computing is brutal arithmetic. A modern core can do tens of floating-point operations in the time it takes to fetch one number from main memory. So if your algorithm streams huge arrays through the processor touching each value only once, the processor spends almost all its time waiting, not computing — the flop count becomes irrelevant and bandwidth is the real bottleneck. Designing fast numerical code is therefore less about counting multiplications and more about arranging the work so that data, once hauled up into cache, gets reused many times before it is evicted. Locality, not flops, is the modern currency.
Reading a value from a register costs about 1 cycle; from L1 about 4 cycles; from L3 about 40 cycles; from main memory about 200-300 cycles. So a single main-memory fetch can cost as much as 100 floating-point multiplies — which is why a program that misses cache constantly can run 10x to 100x slower than the same flop count run from cache.
Each rung down the hierarchy is ~10x bigger and ~10x slower; one DRAM fetch can cost 100 flops.
The hardware manages caching for you, so the hierarchy is mostly invisible — which is exactly the trap. Two algorithms with identical flop counts can differ by 50x in wall-clock time purely from how they walk memory; profiling that ignores cache behaviour will mislead you.