an induction head

If I say 'Mrs. Dalloway said she would buy the flowers herself... Mrs. Dalloway said she would buy the', you instantly expect 'flowers'. You are not reasoning about flowers; you are noticing a pattern repeating and copying its continuation. An induction head is a component inside a transformer that learned to do exactly this little trick: spot that the recent context is repeating an earlier passage, and predict whatever came next last time.

Mechanically it is a beautiful, concrete example of a circuit. It typically works as a pair of attention heads. The current token, say a name's first part, triggers a search back through the text for the previous time that same token appeared; a second head then attends to the token that followed it back then and copies that token forward as the prediction. In short: 'A was followed by B earlier; I just saw A again, so I predict B.' This in-context copying is a foundation of how models pick up patterns from the prompt itself, without any weight changes.

Induction heads matter for interpretability because they are one of the first nontrivial computations that researchers fully reverse-engineered and verified causally, making them a proof-of-concept that real circuits inside transformers can be understood. They are also linked to in-context learning — the model's striking ability to learn a new pattern from examples in the prompt. The caveat is that the textbook two-head story is a clean idealization; in large models the behavior is spread across many heads and mixed with other machinery, so 'the induction head' is a useful abstraction rather than a single tidy gadget.