token streaming

Generation is autoregressive, so tokens become available one at a time over the whole length of a response. Token streaming sends each token to the client the moment it is produced, instead of buffering the entire answer and shipping it at the end. The user starts reading after the time-to-first-token rather than after the full completion, which for a long answer is the difference between a fraction of a second and many seconds, dramatically improving perceived responsiveness even though total generation time is unchanged.

In practice the server pushes incremental updates over a long-lived connection — server-sent events, chunked HTTP, or a websocket — emitting token or text deltas as they are decoded. Some subtleties matter: tokens are sub-word pieces, so the server must handle multi-byte characters that straddle token boundaries to avoid emitting broken glyphs, and it must surface end-of-stream and error states cleanly mid-flight. Streaming pairs naturally with continuous batching, where many concurrent streams are advanced one decode step at a time.

Streaming is purely a delivery and perception optimization: it lowers latency-to-first-word and time-to-first-meaningful-content without changing throughput or the tokens themselves.