multi-head attention

Multi-head attention runs several self-attention computations in parallel and then merges their results. The intuition is that one attention pattern can only express one kind of relationship at a time — a single weighting that, say, groups by colour cannot simultaneously group by shape or by spatial proximity. By giving the model several independent 'heads', each free to learn its own query/key/value projections, you let it attend to several different relationships at once: one head may track texture, another long-range layout, another local edges. The outputs are then combined so the next layer sees all these views together.

Mechanically, the model dimension D is split across h heads. Each head gets its own smaller projections that map tokens to queries, keys and values of dimension d_k = D/h, performs the full scaled dot-product attention within that subspace, and produces a d_k-dimensional output per token. The h head outputs are concatenated back to width D and passed through one final learned output projection that lets the heads mix. Crucially, because each head works in D/h dimensions, the total compute and parameter count is roughly the same as one full-width attention — you get diversity essentially for free, by partitioning rather than adding capacity.

Splitting into subspaces is what makes the diversity possible: each head's projection picks out a different slice of feature space to compare in, so heads can specialize. Studies of trained transformers find heads that behave like edge detectors, heads that attend to the class token, heads that copy positional neighbours, and many heads that turn out redundant and can be pruned with little loss — evidence that the gain is real but unevenly distributed.

In vision specifically, multi-head self-attention (MHSA) is the spatial-mixing core of every transformer encoder block; the number of heads (often 6 to 16 in standard ViTs) is a tuning knob trading representational diversity against per-head dimensionality. Too few heads underuses the parallel-relationships idea; too many leaves each head too low-dimensional to compare meaningfully.