activation patching

To prove a particular wire in a machine matters, the cleanest test is to swap it and see if the output changes. Activation patching is that experiment for a neural network. You run the model on two inputs, then transplant a small piece of internal activity from one run into the other and watch whether the answer flips. If swapping one component's activity hands over the answer, that component was causally carrying the relevant information — not just correlated with it.

A typical setup uses a clean run and a corrupted run. Run the model on 'The Eiffel Tower is in Paris' (clean) and on 'The Colosseum is in Rome' (corrupted), then take the activation at one specific layer-and-position from the clean run and paste it into the corrupted run, leaving everything else corrupted. If that single patch is enough to make the model say 'Paris' again, you have localized where the 'Eiffel-Tower-implies-Paris' information lives. Done across all positions and layers, this builds a causal map of which components matter for the behavior — the gold-standard evidence behind claimed circuits and features.

Activation patching matters because it upgrades interpretability from storytelling to experiment: it is how researchers tell a real mechanism from a coincidence, and it is the test that catches many interpretation illusions. The caveats are subtle but real. Patching shows a component is sufficient or necessary for the effect under your specific clean/corrupted pair, which may not generalize; results depend on how you corrupt the input; and because the network has backup pathways, a component can look unimportant in isolation yet matter when others are also disabled. It is the strongest common evidence we have, used carefully — not an infallible oracle.