the residual stream

Picture a conveyor belt running straight up through the whole model. Each token starts on the belt as its embedding, and every layer, instead of replacing what is there, simply adds its contribution on top. This running total is the residual stream — the central highway along which information travels from the input all the way to the final prediction.

Concretely, each sublayer computes an update and adds it back: the output is the input plus whatever attention or the feed-forward block produced. These residual connections mean gradients can flow straight down the belt during training without vanishing, which is what makes very deep stacks trainable at all. Because everything is added into one shared space, layers can read what earlier layers wrote and write messages for later layers to pick up.

This additive view is the backbone of modern interpretability. The residual stream acts like a shared communication channel with limited bandwidth; heads and feed-forward blocks read from it and write to it in specific directions. Thinking of computation as messages added to this stream, rather than as opaque layer outputs, is what lets researchers trace how a particular fact or behavior is assembled.