the ideal pipeline speedup

Here is the tempting promise of pipelining: split the work into k stages and you go k times faster. Five stages, five times the throughput. It sounds like free money. The honest version is that k is the ceiling you can approach but never quite reach, and several real-world frictions hold you below it.

The ideal arithmetic is simple. If non-pipelined work takes time T, and you cut it into k equal stages, each stage takes T/k, so once the pipeline is full one instruction finishes every T/k — a speedup of k. But three things eat into that. First, the stages are never perfectly equal: the slowest stage sets the clock, so if one stage is fatter than the rest you waste time in the others. Second, every stage gains a pipeline-register delay, so the cycle is a bit longer than T/k. Third, and most important, the pipeline must fill up (the first instruction still takes the full latency before any throughput appears) and it stalls whenever a hazard interrupts the smooth flow.

So the realistic speedup is k divided by a penalty: roughly speedup = k / (1 + stalls per instruction), after accounting for fill time and uneven stages. For a long program the fill time is negligible, but hazards are not — a stream peppered with branches and dependent loads might run at well under half the ideal. This is why simply adding stages does not keep doubling performance: each extra stage buys a faster clock but raises the cost of every stall and misprediction. The art is keeping the line moving, not just making it longer.