multi-head attention
/ MUL-tee-hed uh-TEN-shun /
Multi-head attention runs several attention computations side by side instead of just one, then stitches their results together. Think of a small committee reading the same sentence: one member tracks who-did-what-to-whom, another watches verb tenses, a third keeps an eye on which adjective belongs to which noun. Each "head" is a separate, independent attention with its own learned way of asking questions, and each looks at the sentence through a different lens.
Why split it up? A single attention has to cram all relationships into one set of weights, forcing it to average over patterns that ought to stay distinct. With multiple heads, each can specialize in a narrower kind of relationship. The model first projects each word into several smaller query/key/value sets — one per head — runs attention separately in each, concatenates the outputs, and passes them through one more linear layer to mix the findings back into a single representation. The total compute is kept comparable to one big head by making each head smaller.
In practice multi-head attention is a quiet workhorse: it reliably helps, and it is part of every Transformer. But the tidy story that "head 3 always handles pronouns" is mostly a hopeful simplification. When researchers inspect heads, some do show interpretable roles, many look redundant, and a good number can be pruned away with little loss. The heads are useful in aggregate; assigning each a clean human job is more storytelling than fact.
A model with 8 heads splits a 512-dimensional word vector into eight 64-dimensional slices. Each slice runs its own attention over the sentence, the eight results are concatenated back to 512 dimensions, and a final projection blends them.
One big attention is split into several small parallel ones, then recombined.
More heads is not automatically better. Beyond a point, extra heads tend toward redundancy, and a sizeable fraction can often be removed after training with negligible effect — evidence that the model spreads its work loosely rather than assigning each head an essential, irreplaceable duty.