TTFT and TPOT metrics

LLM serving cannot be summarized by one latency number, because generation has two perceptually distinct phases. Time-to-first-token (TTFT) is how long a user waits after sending a prompt before the first word appears; it is dominated by the prefill pass and by queueing, and it sets how responsive the system feels. Time-per-output-token (TPOT), also called inter-token latency, is the steady cadence at which subsequent tokens stream out; it is dominated by the memory-bound decode step and determines whether the text reads fluidly or stutters.

These two are the core service-level objectives of a serving stack, and crucially they trade off against throughput and against each other. Larger batches and continuous batching raise total tokens-per-second but can lengthen any single request's TPOT; a giant prompt's prefill inflates others' TTFT unless chunked. End-to-end latency for a response of N tokens is well-modeled as TTFT plus N times TPOT, so the two metrics together let operators set and verify SLOs and reason about tail behavior at a given percentile.

Reporting a single mean hides the tail: P99 TTFT and P99 TPOT under realistic concurrency are what actually govern user experience.