attention head roles

When researchers peer inside a trained Transformer, they find that individual heads have quietly specialized. Some become 'previous-token heads' that mostly look one step back; some are syntactic heads that connect verbs to their subjects; some track quotation marks or matching brackets. Nobody assigns these jobs — they emerge from training, like workers on a line drifting into the tasks they happen to be best at.

A celebrated example is the induction head, a pair of heads that together implement a simple copy rule: having seen the pattern 'A B' earlier, they help the model predict 'B' again right after the next 'A.' This kind of circuit is a big part of why LLMs can pick up a format from a few examples in the prompt. Other heads handle positional bookkeeping, deduplication, or routing rare facts.

These roles are real but messy. Many heads are polysemantic, doing several loosely related things; some are nearly redundant and can be pruned with little loss. So 'this head does X' is a useful approximation, not a clean wiring diagram — interpretability work is still mapping which heads matter and why.