time to first token

When you hit enter, there is a pause before the first word streams out. That pause is the time to first token, and it is what makes an assistant feel snappy or sluggish. It is dominated not by raw generation speed but by how long prefill takes plus any time the request spends waiting in a queue.

TTFT covers queueing, the prefill of the whole prompt, and the production of that first output token. It grows with prompt length, because prefill must process every input token, and with server load, because the request may wait for a scheduling slot. Techniques that cut it include chunked prefill that interleaves prefill with ongoing decodes, prefix caching that reuses the KV cache of a shared prompt, and disaggregating prefill onto separate hardware. It is usually reported alongside inter-token latency to describe the full streaming experience.