Why efficiency is now a first-class concern
A model that only runs in a datacenter is a model most people never touch. The frontier of useful AI keeps moving toward the user: a transformer summarizing email on a laptop, a vision model reading a label on a $5 microcontroller, an assistant that answers offline on a plane. That movement is what we mean by on-device inference — and it is governed less by cleverness than by three hard physical budgets: compute, memory, and energy.
Volume I taught you what these models are. This track is about making them fit. The same 7-billion-parameter network that needs 14 GB of memory in 16-bit precision must somehow run inside a phone's 4 GB budget, draw less than a watt, and answer in under a second. Closing that gap of 4× to 100× is the entire discipline of efficient AI, and it is almost never one trick — it is a stack of them, each shaving a different cost.
Weight memory is just parameter count times bytes per number — a 7-billion-parameter model in 16-bit precision already costs 14 GB.
Read the bill: compute vs. memory vs. bandwidth
The single most useful habit is to ask which resource a workload is bound by. Training and large-batch serving are often compute-bound: the accelerator's math units are the bottleneck. But single-user generation — one token at a time — is usually memory-bandwidth-bound: each new token must stream the entire weight matrix from memory, and the math is trivial by comparison. This is why inference cost does not track FLOPs alone, and why latency and throughput can be improved by completely different techniques.
The roofline diagnostic: a workload stays memory-bound until its arithmetic intensity exceeds the device's compute-to-bandwidth ratio.
A taxonomy of every way to make a model cheaper
Almost every method in this track falls into one of four families. It helps to hold the whole map in your head before diving into any one corner:
- Use fewer bits per number — quantization keeps every weight but stores each in 8, 4, or even ~1.6 bits instead of 16, cutting memory and bandwidth proportionally.
- Remove numbers entirely — pruning and sparsity delete weights, channels, or whole heads that contribute little, so the network does less work.
- Re-express the same numbers more compactly — low-rank factorization replaces a big matrix with the product of two thin ones; distillation re-expresses a big model as a small one.
- Spend compute only where needed — early exits, dynamic depth, and token merging skip computation on easy inputs at run time, rather than shrinking the model up front.
These families compose. A production edge model might be distilled, then pruned, then 4-bit quantized, then served with early exits — each step independent, each multiplying the savings. The art is ordering them and knowing when each stops paying off. That is the journey of the next four guides.
Measuring honestly: the only metrics that matter
Compression papers love to report parameter counts and theoretical FLOP reductions. On a real device, neither may move the wall-clock at all. A 50%-sparse model runs no faster unless the hardware has a kernel that exploits the sparsity; an unstructured-pruned matrix is still a dense matrix to a GPU. So the discipline is to measure end-to-end on the target: tokens per second, time-to-first-token, peak memory, and joules per query — not the proxy your paper happens to optimize.
State efficiency as a constrained problem: minimize real cost while staying above an accuracy budget you actually care about.