The middle knob of the iron law
The previous guide gave us the iron law: CPU time = instruction count x CPI x cycle time. Three factors, and a different layer owns each one. The compiler and the ISA mostly set the instruction count; the circuit designers and the clock set the cycle time; and the microarchitecture — the pipeline, the caches, the branch predictor — mostly sets the middle factor, cycles per instruction (CPI). This guide lives inside that middle knob, because it is the one that hides the most and explains the most.
CPI is simply the average number of clock cycles each instruction takes, measured over a whole program: total cycles divided by total instructions. Flip it over and you get IPC (instructions per cycle), the same fact read the other way — a CPI of 0.5 is an IPC of 2, a CPI of 4 is an IPC of 0.25. Both are averages, and that word is doing a lot of work. No single instruction has 'a CPI'; CPI is what falls out when you blend every instruction's true cost together, weighted by how often each kind runs.
Where the cycles actually go
Here is the picture that makes CPI honest. Imagine an ideal pipelined processor that finishes one instruction every cycle — a perfect CPI of 1, the dream the five-stage pipeline was sold on. Real CPI is that ideal floor plus the cycles wasted whenever the machine cannot keep the line moving. Every stall, every bubble, every wait adds onto the base. So effective CPI is best read as a sum of penalties stacked on top of an ideal, and the whole craft of performance is shrinking the penalties.
Where do those extra cycles come from? Almost all of them are ghosts from the earlier rungs of this ladder. A load-use hazard freezes the pipeline for a cycle when an instruction needs a value the load just before it has not finished fetching. A branch that the branch predictor guesses wrong triggers a flush, throwing away the wrongly-fetched instructions and costing several cycles. And the heaviest one by far: a cache miss can stall the core for hundreds of cycles while it waits on slow DRAM. Each of these contributes a slice of CPI proportional to how often it happens times how many cycles it costs.
Effective CPI = ideal CPI + sum of (event rate x event penalty) Worked example (1 GHz core, ideal pipelined CPI = 1.0): source frequency (per instr) penalty CPI added ---------- --------------------- ------- --------- ideal base -- -- 1.00 branch mispredict 2% of instrs 15 cyc 0.30 L1 cache miss 5% of instrs 12 cyc 0.60 L2 cache miss 1% of instrs 100 cyc 1.00 ---------------------------------------------------------- effective CPI 2.90 So the 'ideal' 1.0 machine actually runs at CPI 2.90 -- it spends about 66% of its cycles NOT making forward progress. The cache misses alone cost more than the useful work. THIS is 'where the cycles go'.
Stare at that table and the lesson reframes everything. The 'one instruction per cycle' machine actually averages 2.9 cycles per instruction, and the single biggest culprit is not arithmetic — it is the memory wall we met two rungs ago. This is why architects obsess over caches and branch prediction rather than faster adders: the adder was never the bottleneck. It is also the deep justification for making the common case fast — you attack whichever penalty owns the most cycles, and on most programs that is the memory system.
Decomposing CPI by instruction mix
There is a second, complementary way to slice CPI: not by stall source, but by instruction type. Different classes of instruction genuinely take different numbers of cycles. A register-to-register add is cheap; an integer multiply takes several cycles; a floating-point divide takes many; a load that misses in cache takes hundreds. The program's overall CPI is the weighted average of these per-class costs, where the weights are how much of the program is each class — its instruction mix.
- List the instruction classes in your program (say: ALU ops, loads, stores, branches) and the fraction of executed instructions each one represents — that is the instruction mix.
- For each class, find its average cycle cost on this microarchitecture, including the stalls it tends to trigger — a branch carries its mispredict tax, a load carries its miss tax.
- Multiply each class's frequency by its cycle cost, then add the products together. That weighted sum is the program's overall CPI — the same number the hardware counters would report.
- Now you can target your effort: the class with the largest frequency-times-cost product is where optimising pays off most. Halving a class that owns 2% of cycles barely moves the needle; halving the one that owns 60% transforms the program.
These two decompositions — by stall source and by instruction mix — are the same total cycles counted two ways, and good performance work uses both lenses. The mix view tells you what kind of work dominates; the stall view tells you what is stopping that work from flowing. Together they answer the question this guide is named for: when you ask where the cycles go, you can name them, one penalty and one instruction class at a time, instead of waving at a mysterious 'it feels slow'.
Why MIPS and FLOPS mislead
Now we can dismantle a tempting but treacherous metric. MIPS — millions of instructions per second — sounds like a clean speed rating: bigger is faster, surely? But MIPS equals clock rate divided by (CPI times one million), so it counts instructions retired, not work done. And instructions are not work. Two machines solving the identical problem can report very different MIPS simply because their ISAs chop the job into different numbers of instructions.
Worse, MIPS can move the wrong way. A compiler optimisation that replaces a slow instruction sequence with a faster one usually shrinks the instruction count — so the program finishes sooner yet its MIPS rating drops, because it now retires fewer instructions per second even though each second does more real work. A metric that falls when you genuinely speed up the program is not measuring performance; it is measuring instruction churn. The honest jokes write themselves: MIPS, 'Meaningless Indicator of Processor Speed'.
FLOPS — floating-point operations per second — is the same trap wearing a lab coat. It is more useful than MIPS because a floating-point op is closer to actual scientific work, which is why it rules supercomputer rankings. But it still counts a kind of operation, not the time to solve your problem, and it is gamed routinely: a kernel can hit a dazzling peak FLOPS while computing something you do not need, or while starved for memory bandwidth so that the floating-point units mostly idle. The recurring lesson is the megahertz myth from the last guide, generalised: any metric that counts a proxy — hertz, instructions, flops — can be inflated without making your program finish sooner.
So which metric is right?
Here is the honest synthesis, and it is the spine of this whole rung: there is no single correct performance number, because the right metric depends on what you actually care about. If you care about how long one job takes, you want execution time — the iron law's answer — and CPI is a tool for understanding it, not a goal in itself. If you serve millions of requests, you may care about throughput or tail latency far more than any single job's CPI. If your phone is in your pocket, you may care about energy per task above all, which the next guides take up.
Hold this together and the iron law stops being a formula and becomes a map. Instruction count, CPI, and cycle time partition every speedup, and CPI in turn partitions into an ideal floor plus a stack of nameable penalties. When someone says a chip is 'faster', you now know to ask: faster at what, measured how, and which of the three factors moved? That refusal to be fooled by a single shiny number is exactly what the next guide builds into a discipline — benchmarking without fooling yourself.