induction heads

Induction heads are a specific attention circuit, discovered through mechanistic interpretability, that lets a transformer continue a pattern it has just seen. If the context contains A B somewhere and then A again, an induction head notices the earlier A, looks at what followed it (B), and raises the probability that the next token is B. In plain terms it implements the rule: this happened before; what came next? Predict that again.

The circuit is built from two heads working in composition across layers. A previous-token head in an earlier layer writes, into each position's residual stream, information about the token that preceded it. A later induction head then forms a query from the current token and matches it against those previous-token signatures, so it attends to the position right after the earlier occurrence of the current token and copies that token forward. This prefix-matching-and-copying is enabled by query/key composition through the residual stream.

Induction heads matter because they emerge during training in a sharp phase change that coincides with a sudden jump in in-context learning ability, strong evidence that they are a primary mechanism behind it. They are a flagship result of the circuits research program, showing that interpretable algorithms can be reverse-engineered from learned weights.