induction heads

Induction heads are the first piece of an LLM that was reverse-engineered cleanly, and they do something delightfully simple: they continue patterns by copying. If a sequence contained the bigram "Harry Potter" earlier and the model now sees "Harry" again, an induction head looks back to the earlier "Harry," notices what came next, and predicts "Potter." It implements the rule: I have seen this before; here is what followed.

Mechanistically it is a two-head circuit. An earlier previous-token head writes, onto each position, information about the token before it. A later induction head then forms a query that searches for an earlier position whose stored predecessor matches the current token, attends there, and copies the following token forward. The two heads compose through the residual stream into a working copy-the-pattern algorithm that no single head could do alone.

Induction heads matter beyond their own trick. They emerge during training at a sharp phase change that coincides with a jump in the model's ability to learn from context, and many researchers see them as a primitive ingredient of in-context learning itself. They are the canonical proof that real, nameable algorithms can be found inside transformers.