activation patching

Activation patching is interpretability's version of a controlled experiment. To test whether some internal component causes a behavior, you run the model on two inputs — a clean one and a corrupted one that breaks the behavior — and then surgically copy one activation from the clean run into the corrupted run. If pasting that single piece back restores the correct output, you have localized a part of the computation that matters; if nothing changes, that part was a bystander.

The power is that it tests causation, not mere correlation. Probing or attention maps can tell you a component contains relevant information; patching tells you the component is actually used to produce the result. By sweeping the patch across layers and positions you draw a causal map of where the relevant information lives and flows. Variants like causal tracing, path patching, and attribution patching trade precision for speed in large models.

The pitfalls are real. Results depend on how you corrupt the input, components can compensate for one another so a true contributor looks unimportant, and patching off-distribution activations can produce artifacts. It is the most trusted tool in the mechanistic toolkit precisely because it intervenes, but it still requires careful controls to interpret.