throughput versus latency in a pipeline

Picture a car wash with a long tunnel. The tunnel takes ten minutes to drive through, and that ten minutes is the same whether it is empty or busy. That is latency — how long one car takes from entrance to exit. But if a new car can enter every minute while ten cars are inside the tunnel at once, the wash finishes one car per minute. That rate — cars finished per minute — is throughput. The two numbers are different, and a pipeline improves one without improving the other.

Latency is the time for one instruction to travel from start to finish; throughput is the rate at which finished instructions come out the end. In a non-pipelined machine, latency and throughput are tied together: if an instruction takes 5 nanoseconds, you finish one every 5 nanoseconds. A pipeline cuts that tie. Split the 5 nanoseconds into five 1-nanosecond stages: each instruction still takes about 5 nanoseconds to pass through (latency is unchanged, even slightly worse), but once the pipe is full a new instruction finishes every 1 nanosecond. Throughput went up roughly five times while latency stayed put.

Beginners must hold both ideas at once, because they answer different questions. If you care how soon a single answer is ready — say, one critical instruction a later one is waiting on — latency rules, and pipelining did not help that. If you care how much total work gets done per second — a long stream of independent instructions — throughput rules, and pipelining is a huge win. Real programs usually run long streams of work, which is why pipelining pays off; but a chain of dependent instructions exposes the latency and stalls the pipe.