tokens per second

Once words start streaming, tokens per second is how fast they keep coming. For a single user it sets the reading-speed feel of the answer; for a whole cluster, the aggregate version measures how much total work the system is getting done. The two numbers are easy to confuse and often very different.

Per-request decode speed is set mainly by memory bandwidth: each token requires re-reading the model's weights, and the KV cache, from GPU memory, so tokens per second is roughly the memory bandwidth divided by the bytes read per token. That is exactly why quantization and smaller KV caches speed decode up, and why decode is called memory-bound. System-wide throughput — total tokens per second across a full batch — is a separate figure that can be far higher, because the weight read is amortized across the batch, so always check whether a quoted speed is per request or aggregate.