latency versus throughput
Picture a coffee shop. Latency is how long YOU wait from ordering to getting your cup. Throughput is how many cups the shop serves per hour across all customers. These are different questions, and improving one can hurt the other: a shop could batch ten orders to brew them efficiently (great throughput) while you stand there longer (worse latency). Almost every performance discussion in systems is really about which of these two you are optimizing.
Precisely: latency is the time for ONE operation to complete, measured per request (milliseconds, microseconds). Throughput is the RATE of completed operations, measured per unit time (requests per second, bytes per second, also called bandwidth). They are not reciprocals, and they trade off in subtle ways. Batching and pipelining raise throughput by amortizing fixed costs over many items, but they add waiting and so raise latency. Conversely, doing each request the instant it arrives keeps latency low but may waste capacity. The relationship between them under load is captured by Little's law: the average number of requests in flight equals throughput multiplied by latency (L = throughput x latency). A consequence is that as you push throughput toward the system's maximum, queues build and latency rises sharply — so the highest-throughput operating point usually has terrible latency, and you deliberately run below peak to keep latency acceptable.
It matters because choosing the wrong target wastes effort and can make users miserable. A batch analytics job cares about throughput (finish the night's data by morning) and barely about the latency of any single record; an interactive editor cares about latency (keystroke-to-screen must feel instant) far more than aggregate throughput. The honest caveat: do not collapse them into one 'speed' number. A change that doubles throughput while tripling tail latency is a clear win for one workload and a clear loss for another — name which metric you are optimizing before you start, and watch the other one so you do not silently wreck it.
Single request: 2 ms latency. Batch 64 requests before processing: throughput up ~3x (fixed costs amortized), but each request now waits for the batch -> latency ~10 ms. Little's law: 500 req/s x 0.010 s = 5 requests in flight on average.
Batching trades latency for throughput; Little's law links in-flight count, rate, and latency for a system in steady state.
They are not opposites and not interchangeable: 'fast' is ambiguous until you say latency or throughput. Near saturation, small throughput gains cost large latency increases as queues form — peak throughput and good latency rarely coexist.