a multilevel cache

There is a tension in one cache: you want it tiny so it can be answered in a single cycle, but you also want it huge so it almost never misses. You cannot have both in one structure. The resolution is to stop choosing and instead stack several caches: a tiny, blazing-fast one right next to the core, backed by a larger, somewhat slower one, backed by an even larger, slower one. A multilevel cache is this stack — typically L1, L2, and L3 — each level trading some speed for more capacity than the level above.

On an access, the core checks L1 first; if L1 misses, it checks L2; if L2 misses, L3; and only if L3 misses does the request go to main memory. Each level catches a share of the misses that escaped the level above, so the average miss penalty seen by L1 is small — most L1 misses are caught quickly by L2, not by a long trip to DRAM. L1 is usually split into separate instruction and data caches (a Harvard-style split at the top so the core can fetch an instruction and a load a data value in the same cycle), while L2 and L3 are usually unified. A key design choice is inclusion: an INCLUSIVE hierarchy keeps copies of L1's contents also in L2 (simpler coherence, but wastes capacity duplicating data), while an EXCLUSIVE hierarchy keeps each block in only one level (more effective total capacity, more complex bookkeeping).

Multilevel caching is how the AMAT formula's recursion pays off: L1's miss penalty is just L2's AMAT, and so on, so adding a middle level dramatically lowers the effective penalty without slowing the fast L1. It is the standard organization in every modern CPU. The honest nuances: more levels add latency to a true all-the-way-to-memory miss (you check several caches first) and consume area and power, and on multicore chips the last-level cache is usually shared among cores, which entangles it with coherence and contention concerns beyond a single core's view. Still, for the single-core memory-speed problem, layering caches is the central, decisive technique.