mechanistic interpretability

A trained neural network is a little like a vast city seen from a plane at night: millions of lights (numbers) blinking in patterns, and somewhere in there it has learned to translate French or write code — but you cannot see how it does it. Mechanistic interpretability is the project of landing the plane, walking the streets, and figuring out what each building actually does. It tries to reverse-engineer the computation hidden in a model's weights back into parts a human can name, draw, and check.

The word mechanistic signals that you are not satisfied just knowing the model is accurate; you want the actual algorithm it runs inside itself. Researchers do this by studying the model's internal activations (the numbers flowing through it as it processes a sentence), proposing that some direction in that activity stands for a human-meaningful concept — a feature — and that features wire together into small computational subgraphs — circuits. As a real example, researchers found small components called induction heads inside transformers that carry out a concrete rule: having once seen the pattern A then B, when A appears again, predict B. That is a copy-the-pattern algorithm you can actually point to.

It matters for safety because behavior tests only tell you what a model did on the cases you happened to try; a mechanism can tell you why, and might reveal a hidden shortcut, a brittle heuristic, or even a concealed goal before it causes harm. But the field is young and humbling. The internal parts are tangled — one neuron can take part in many unrelated computations — and a tidy explanation can fit the observations without being the true mechanism. Mechanistic interpretability today is best read as a fast-growing collection of partial wins, not a finished microscope.