streaming generation

Because an LLM produces text one token at a time, it does not have to wait until the whole answer is finished to show you anything. Streaming generation sends each token to the screen the moment it is decoded, which is why a chatbot's reply types itself out word by word. Nothing about the model changes; what changes is the delivery — pieces are pushed to you continuously rather than held back and handed over in one block at the end.

The win is felt rather than measured: time-to-first-token drops to a fraction of a second, so the experience feels alive even when the full response takes many seconds. It also lets a user read along, interrupt early, or cancel a wrong turn before paying for the rest. The trade-offs are engineering ones — you cannot validate or reformat output you have already shown, post-hoc constraints get harder, and partial tokens (a half-formed word or emoji) need careful buffering so the stream reads cleanly.