mechanistic interpretability

Mechanistic interpretability treats a neural network like a strange piece of hardware that someone forgot to document, and tries to reverse-engineer it. Rather than asking only which inputs change the output, it asks for the algorithm: which specific weights and activations implement a given behavior, and how they wire together into a step-by-step computation. The aspiration is to recover something you could in principle reimplement by hand.

The unit of analysis is the circuit — a small subgraph of features and the attention heads or neurons connecting them that together carry out one task, like detecting that a token was seen earlier and copying it forward. Researchers find these by patching activations, ablating components, and decoding internal directions, then checking that the proposed mechanism actually drives the output. Success looks like a faithful, predictive story: change the circuit, change the behavior exactly as the story says.

It is the most ambitious branch of interpretability and the most fragile. Real circuits are messy, features overlap in superposition, and a clean toy result rarely scales unchanged to a 100-billion-parameter model. Still, it has produced genuinely surprising finds — induction heads, copy-suppression, repeated motifs across models — that feel less like storytelling and more like discovering the model's actual gears.