Two different questions, easy to confuse
The previous guide left you with a sharp habit: never collapse measurements to one number until you have looked at their shape, because the tail of the distribution is often the real story. Now we name the two questions that shape was answering all along, because beginners blur them constantly. Latency is how long one operation takes — the time from request to response for a single call. Throughput is how many operations finish per unit time — requests per second, bytes per second, frames per second. They sound like two ways of saying the same thing, and they are not; they can move in opposite directions, and optimizing one can wreck the other.
A small picture nails the difference. Imagine a checkout with one cashier who takes exactly 30 seconds per customer. The latency each customer feels is 30 seconds, and the throughput is 2 customers per minute. Now open ten lanes. Each customer still waits 30 seconds — latency is unchanged — but the store now clears 20 customers per minute: throughput went up tenfold while latency stayed flat. The two are not locked together. You raised one by adding parallel capacity without touching how long any single transaction takes. This is the heart of the latency-versus-throughput distinction: throughput is often a property of the system's width, latency a property of the single path through it.
But push the system harder and the two start to interact, in a way that surprises people. Keep feeding more customers than ten lanes can clear and a queue forms; now each customer's latency is no longer 30 seconds but 30 seconds plus however long they waited in line. As you approach maximum throughput, latency does not stay flat — it climbs, then explodes. This is queueing in action, and it is why you can never read a latency number without asking "at what load?" A service that responds in 1 ms when idle can take 200 ms at 90% of its throughput ceiling. The same code, the same machine — the only thing that changed is how full the pipe is.
Why the average is the wrong summary
Once latency is the question, the average becomes actively misleading — and this is where the warning from the last guide cashes out. Suppose a request takes 1 ms in the common case, but once every hundred calls it hits a slow path — a heap allocation that triggers a page fault, a cache-cold lookup, a brief context switch to another thread — and takes 100 ms. The mean is about 2 ms, which looks great and describes no actual request: requests are either fast (1 ms) or slow (100 ms), never the 2 ms the average reports. The mean has smeared a bimodal reality into a single number that hides exactly the cases you care about.
So latency is reported with percentiles, not a mean. The p50 (the median) is the value half your requests beat. The p99 is the value 99% of requests beat — equivalently, the threshold the slowest 1% blow past. The p999 (the 99.9th percentile) is the slowest one in a thousand. Reading them together tells the real story: a service might run p50 = 1 ms, p99 = 12 ms, p999 = 110 ms. That spread between the median and the high percentiles is the tail, and it is the part a single average can never show you. The slang is exact: p99 is "the latency your unluckiest one-in-a-hundred users see."
Why the tail matters more than it looks
It is tempting to wave the tail away — "only 1% of requests are slow, who cares?" The reason to care is that real requests rarely travel alone. Modern systems fan out: one user click triggers a page that calls twenty backend services in parallel, and the page is not done until the slowest of those twenty replies. If each backend has a clean p99, the chance that all twenty land in their fast 99% is 0.99 to the twentieth power — about 0.82. So roughly 18% of user-facing requests touch at least one backend's slow tail. The rare per-service tail has become the common user experience. This is the amplification that makes tail latency the metric that actually governs how a distributed system feels.
P(page is fast) = (p99 fast fraction) ^ (number of backends) 1 backend : 0.99 ^ 1 = 0.99 -> 1% of pages hit a slow tail 20 backends: 0.99 ^ 20 = 0.818 -> ~18% of pages hit a slow tail 100 backends: 0.99 ^ 100 = 0.366 -> ~63% of pages hit a slow tail
There is a second reason the tail outweighs its frequency: the slow path often is the bug. A p50 of 1 ms tells you the code is capable of being fast; the p999 of 110 ms tells you something occasionally goes very wrong, and that something is usually a concrete, fixable mechanism — a lock held too long, a hot-path allocation that occasionally triggers the allocator's slow path, a garbage-collection pause, a cache that periodically goes cold, a context switch at the wrong moment. The tail is where your worst-behaved code paths announce themselves. Chasing the p999 down is not polishing; it is debugging the rare catastrophe your average was politely hiding from you.
Coordinated omission: how your tail measurement lies
Now the most important and least obvious idea in this guide, and the one that makes most homegrown latency numbers worthless. Picture the standard load-testing loop: send a request, wait for the reply, time it, immediately send the next. It looks honest. But watch what happens when the server stalls for 100 ms — a long allocation, a GC pause, whatever. Your loop is blocked waiting for that one slow reply, so during those 100 ms it sends no new requests at all. You wanted to fire one request every millisecond; instead you sent one slow one and recorded a single 100 ms sample. The 99 requests you were supposed to send during the stall — every one of which would also have been slow, because the server was down the whole time — were never sent and never measured.
This is coordinated omission: your measurement client coordinates with the very stalls it is trying to measure, and omits exactly the slow samples it should have captured. The effect is devastating to the tail. You recorded one 100 ms outlier when reality contained a hundred slow requests; your histogram shows the stall as a 1-in-N rarity when in truth it dominated a 100 ms window. The median looks fine, the p99 looks reassuring, and the system that produced them would fall over in production. Coordinated omission is not a rounding error — it routinely understates the true tail by ten times or more, and it does so silently, with numbers that look perfectly respectable.
The fix is to measure latency against the schedule a request should have been sent on, not the schedule a stalled client actually managed. If you intended to send one request per millisecond starting at a fixed clock, then a request that was supposed to go out at time t but only got sent at t + 50 ms has already suffered 50 ms of latency before the server even saw it — and that waiting time must be counted. Good load generators (and good benchmark frameworks, in the spirit of reproducible benchmarking) do exactly this: they fix the intended send times in advance and charge each request its full delay from when it should have started. The practical rules below keep you honest.
- Fix the intended send schedule up front (for example one request every millisecond) instead of sending the next request only after the previous reply arrives.
- Measure each request's latency from its intended start time, not from when a stalled client finally managed to send it — count the queueing delay the stall imposed.
- Use a load generator that corrects for coordinated omission (or a closed-loop model only if your real clients truly block on each reply, like a single-threaded CLI).
- Report percentiles (p50, p99, p999) at a stated, fixed load, never the mean and never an unstated load — a latency number without a load is meaningless.
Tuning the right knob
Because latency and throughput are different questions, they call for different fixes — and a change that helps one can hurt the other, so you must know which you are optimizing before you touch anything. Batching is the classic example. Grouping many small operations into one big one amortizes fixed per-operation overhead — one syscall instead of a hundred, one allocation instead of many — which raises throughput, sometimes dramatically. But a request that must now wait to fill a batch sits idle until its batch is ready, so batching raises latency for the individual request. Throughput up, latency up: a genuine trade, not a free win. Whether it is the right call depends entirely on whether your users feel each request's latency or only the aggregate rate.
Adding parallelism — more threads, more cores, more lanes — is the move that raises throughput while leaving single-request latency roughly alone, exactly the ten-checkout-lanes picture from the opening. But this is precisely where the next guide picks up, because parallelism does not scale forever: the serial fraction of your work caps the speedup no matter how many cores you throw at it, which is Amdahl's law. Worse, parallelism can create tail latency it did not have before — contention on a shared lock, a thread that loses its core to a context switch at the wrong moment, the noise that grows as more threads compete for caches and memory bandwidth. Throughput tuning and tail behavior are not independent; widening the system can lengthen the tail.
Carrying it forward
Pull the thread together. Latency and throughput answer different questions — how long for one, how many per second — and they can move independently or even fight, so the first act of any performance work is to name which one matters here and at what load. For latency, the average is a lie that smears a bimodal reality into a number describing no real request; report percentiles and read the gap between p50 and p999, because that gap is the tail. The tail matters far beyond its frequency: fan-out turns a rare per-service slowdown into the common user experience, and the slow path is usually a concrete bug worth hunting. And above all, distrust your own tail measurement: coordinated omission silently erases the slow samples a stalled client failed to send, understating the tail by an order of magnitude with numbers that look respectable.
The opening picture already pointed at the next rung in the argument. Adding lanes lifted throughput without touching latency — until the queue formed and latency climbed. That "until" is the whole subject of the next guide. When you reach for parallelism to raise throughput, how much speedup can you actually buy, and where does the curve flatten? The answer is not "as much as you want": the serial part of your work sets a hard ceiling, and there is a precise law that tells you where it is. The next guide makes that law — Amdahl's law, and its more hopeful cousin Gustafson's — concrete, so that the throughput you chase here meets the limit that governs it there.