multi-head attention

One attention pattern can only emphasize one kind of relationship at a time — but language needs many at once. Multi-head attention solves this by running several attentions side by side, each with its own queries, keys, and values. One head might track grammatical subjects, another might link pronouns to their referents, another might watch nearby punctuation, all in the same layer and all at the same time.

The model splits each token's representation into several lower-dimensional slices, one per head. Each head computes its own scores, softmax weights, and value blend independently, looking at the sequence through its own lens. The per-head outputs are then concatenated back together and passed through a final linear projection that mixes them into a single result the next layer can use.

Splitting rather than duplicating keeps the total cost about the same as one big head while buying far more expressive power. Heads do not coordinate explicitly; they simply learn complementary jobs because the training signal rewards covering different relationships. Modern variants share keys and values across heads to shrink the inference cache.