context mixing

Context mixing is the name for what attention actually accomplishes: it is the step where information held at different positions gets blended together. Going into an attention layer, each token mostly knows about itself; coming out, each token has folded in pieces of every other token it found relevant. A pronoun absorbs a little of its antecedent; a verb absorbs its subject; an ambiguous word absorbs the clues that disambiguate it.

The mixing is precisely a weighted sum of value vectors, with the attention weights deciding the recipe. This cleanly divides the labor inside a block: attention moves and combines information across positions, the mixing, while the feed-forward network transforms each position on its own, no mixing. Stacking blocks lets information hop further and further — a fact two tokens away can reach a token many layers later by being relayed through intermediate positions.

Framing the model this way — mix across tokens, then process per token, repeat — is a useful mental model and a research tool. Measuring how much each input token contributes to a later representation is one way to attribute a model's output to its inputs, though such attributions are approximate and should be read with care.