The clean story that was wrong
Here is a scene that has played out in real interpretability labs. A researcher scrolls through the examples that make one neuron fire hardest, and they are all dogs — golden retrievers, beagles, a husky in snow. 'Found it,' she thinks: the dog neuron. It is a clean, satisfying story. Then someone runs the same neuron over a wider slice of text and images, and it lights up just as eagerly for close-up fur textures, for the word 'loyal', and for a photo of a mop. The tidy label was never the mechanism; it was an artifact of the handful of examples she happened to look at first. Our brains are relentless pattern-finders — we see faces in clouds and intentions in random noise — and that same instinct is exactly what makes reading the inside of a neural network so treacherous.
By the time you reach this guide you have a real toolkit. You have seen features as the model's internal vocabulary, circuits as the wiring that connects them, the superposition hypothesis explaining why meanings are packed together, sparse autoencoders for pulling them apart, and probes and activation steering for reading and nudging them. Each is powerful. But every one of them can produce a result that looks like understanding without being understanding. The name for that failure is an interpretation illusion: a tidy, convincing account of what a part of a model is doing that fits the evidence you happened to examine, yet is not the true mechanism. This final guide of the rung is about the skill that makes all the others trustworthy — telling a genuine interpretation from a seductive illusion.
Plausible is not faithful
Start with a distinction that quietly organizes this whole subject. Imagine a charming tour guide standing before an old building. He spins a vivid story: this scorch mark is from a famous fire, that worn step was hollowed by a king's daily walk. The story is wonderful, internally consistent, and completely convincing — and entirely made up. A good explanation can be plausible (it satisfies a human listener) without being faithful (it accurately reflects what actually happened). Researchers Jacovi and Goldberg drew exactly this line for machine learning in 2020, and it is the single most useful idea for evaluating interpretations: plausibility measures how convincing an explanation is to us; faithfulness measures whether it matches the real computation. The two can come apart completely.
Why does interpretability slide toward plausibility so easily? Because most of the evidence we gather first is correlational and hand-picked. You look at what a neuron responds to and write down a label; you train a probe and find a concept is decodable; you generate a feature visualization that looks like a dog. Each of these tells you that some pattern is associated with some part of the model — but association is not mechanism. The neuron's label may be an artifact of your dataset; the probe may be reading information the model never acts on; the visualization may be a compelling picture that misrepresents what actually drives the unit. In every case a plausible narrative outran the actual cause. An interpretation illusion is just plausibility that we mistook for faithfulness.
A famous skirmish makes the danger concrete. When attention mechanisms became standard, it was tempting to read a model's attention weights as its explanation — 'it focused here, so this word mattered.' In 2019 Jain and Wallace published 'Attention is not Explanation,' showing you could often find very different attention patterns that produced the same prediction, so the weights could not be the reason. Within months Wiegreffe and Pinter answered with 'Attention is not not Explanation,' arguing the test was too harsh and that attention carries real information under stricter conditions. The lasting lesson is not who won; it is that a feature which looks made for explanation — attention literally highlights words — can still fail to be a faithful account, and settling the matter took careful adversarial experiments, not intuition.
From correlation to cause
If association is the trap, causation is the way out. The intuition is the same one an electrician uses: to prove a wire matters, cut it and see if the light goes off. In interpretability the equivalent move is intervention — don't just observe a component, reach in and change it, then watch whether the behavior changes the way your hypothesis predicts. This is the dividing line between a story and an experiment. A neuron correlated with 'dog' is a hint; a neuron you can switch off to make the model stop saying 'dog,' and switch on to make it say 'dog' unprompted, is evidence.
Three causal tools do this work, each tightening the screw. Ablation is the bluntest: zero out or corrupt a component and measure how much the behavior breaks — if removing your candidate circuit leaves the behavior intact, your story was wrong. Activation patching is sharper: run the model on a 'clean' input and a 'corrupted' one, then transplant a single internal activation from the clean run into the corrupted run and see if the answer flips back. If pasting in just that one piece restores the right answer, you have localized where the information actually lives, not merely where it correlates. And activation steering runs the test in the positive direction: add a candidate feature direction to the running activations and check that behavior moves as predicted. When Anthropic amplified a single 'Golden Gate Bridge' feature in 2024 and the model began steering every conversation toward the bridge, that was not a story about the feature — it was a causal demonstration that the direction really carried that concept.
Single interventions test one wire; testing a whole explanation needs more. Causal scrubbing, proposed by Redwood Research in 2022, formalizes the idea. Your hypothesis claims certain activations are interchangeable (they only carry information X) and certain paths don't matter. Causal scrubbing puts that claim on trial: it resamples or swaps activations in every way the hypothesis says should be harmless, and checks whether the behavior survives. If your explanation is faithful, the model keeps working under all the swaps it permits; if you quietly relied on something your story said was irrelevant, performance drops and the hypothesis is falsified. This flips interpretability from 'here is a plausible diagram' to 'here is a claim that makes predictions and could have failed' — the posture of a real experiment.
The probing trap
Probing deserves its own warning, because it is the friendliest tool and the easiest to over-trust. Recall a linear probe: you freeze the model, collect its activations on labelled examples, and train a tiny classifier to predict some property — past tense, sentiment, toxicity. High probe accuracy feels like proof the model 'knows' the concept. But two quieter questions hide underneath, and beginners routinely miss both: did the probe learn the concept from the model, or from its own capacity? And even if the information is in there, does the model ever use it?
The first question was answered crisply by Hewitt and Liang in 2019 with control tasks. Alongside the real task, they trained the same probe to predict random labels assigned to each word. A powerful probe can fit even those random labels — which means high accuracy might reflect the probe's own muscle, not the model's representation. Their fix is selectivity: real-task accuracy minus control-task accuracy. A probe that nails the real concept but cannot fit the random one is genuinely reading the model; a probe that scores high on both is mostly memorizing. The discipline is to prefer simple probes and report selectivity — otherwise you can 'discover' structure that lives in your classifier, not in the network.
The second question — does the model use the information? — is the deeper one, and it is where many probing claims quietly fail. The clean demonstration is amnesic probing (Elazar and colleagues, 2021). Instead of just decoding a property, they surgically remove it from the activations and then watch the model's actual predictions. The striking finding: you can often decode a property with a great probe, yet erasing that property barely changes the model's behavior — meaning the model was not relying on it to decide anything. Decodability and use are different things. A probe that reads 'this model represents whether the user is angry' tells you the information is present; only an intervention tells you whether the model acts on it. For safety this gap is the difference between 'the model encodes the concept of deception somewhere' and 'the model uses a deception process to choose its outputs' — and only the second should reassure or alarm you.
Two illusions on the record
These are not just theoretical risks; the literature has documented illusions in the wild. The result that gave the phenomenon its name is Bolukbasi and colleagues' 2021 study, bluntly titled 'An Interpretability Illusion for BERT.' They examined directions in BERT's activation space — the kind of directions a sparse autoencoder or probe might hand you as a clean feature. On one dataset a given direction looked like it crisply encoded a single coherent concept. But run the very same direction on a different dataset and it appeared to encode an entirely different, equally coherent concept. The apparent meaning was partly a property of the data you used to interpret it, not solely a property of the model. The lesson lands hard on today's tools: a sparse-autoencoder feature that looks beautifully monosemantic on the dataset you inspected might wear a different face on data you did not.
The second case strikes even closer to safety, because it concerns the explanations a model gives about itself. When a model 'shows its work' through chain-of-thought reasoning, it is tempting to treat that written reasoning as a window into why it answered as it did. Turpin and colleagues tested this in 2023 ('Language Models Don't Always Say What They Think'). They secretly biased the input — for example, reordering a multiple-choice prompt so that the correct answer was always option (A) in the worked examples — and the model's answers shifted toward the bias. But its chain-of-thought never mentioned the reordering. Instead it produced a fluent, plausible-sounding justification for the biased answer, confabulating a rationale that had nothing to do with the real cause. The stated reasoning was plausible and unfaithful at the same time.
PROMPT (biased): 6 worked examples whose correct answer is "(A)" in ALL of
them, then a new question whose best answer is really (B).
MODEL CHAIN-OF-THOUGHT (what it SAYS):
"Let's think step by step. Option (A) matches the description given,
the reasoning supports it, so the answer is (A)."
-> a clean, sensible-looking rationale for (A)
ACTUAL CAUSE (what MOVED it):
the (A),(A),(A),(A),(A),(A) ordering in the examples -- never mentioned.
CONTROL TEST: remove the bias (shuffle the examples) -> model answers (B).
Conclusion: the chain-of-thought was a post-hoc story, not the mechanism.Hold both the weight and the limits of these results. They do not show interpretability is hopeless, and they do not show models are scheming — the chain-of-thought confabulation looks more like ordinary human post-hoc rationalization than deliberate deception. Newer work (including a 2025 Anthropic study on reasoning models) finds that even models trained to reason at length often omit a decisive hint from their stated reasoning, so the faithfulness gap is real and not unique to one setup. The flip side is genuinely encouraging: when researchers do apply causal tests, they sometimes confirm a clean mechanism — the reverse-engineered induction head survived rigorous causal checks, and the Golden Gate steering result held up. The point is not despair; it is that the verdict comes from intervention, not from how good the story sounds.
A field guide to spotting an illusion
Put it together into a working habit. Before you trust an interpretation — your own or one in a paper — run it past a short checklist. None of these tests is magic, and passing them is evidence rather than proof, but together they separate most real mechanisms from most seductive stories.
- Causality, not correlation: when you intervene — ablate the component, patch its activation, or steer along the direction — does the behavior change the specific way your hypothesis predicts? If nothing moves, the story is decoration.
- Necessity and sufficiency: check both directions. Is the component needed (remove it and the behavior breaks) and enough (supply it and the behavior appears)? A real mechanism usually passes both; a mere correlate passes neither cleanly.
- Specificity and off-distribution: does the explanation still hold on examples you did not cherry-pick — a broader dataset, held-out cases, adversarial inputs? The BERT illusion lived precisely in the gap between one dataset and another.
- Controls: would a random direction, a random neuron, or a random label score just as well? Report selectivity for probes, and compare against a sensible baseline. If noise does as well as your finding, your finding is noise.
- Faithfulness under the whole hypothesis: where you can, run causal scrubbing — does the behavior survive every swap your explanation says is harmless? An explanation that makes risky predictions and survives them earns far more trust than a pretty diagram.
A few traps catch almost everyone at first. The biggest: treating a label as a discovery. Naming a neuron 'the dog neuron' from its top activations is a hypothesis, not a finding — the work starts when you test it. Second, mistaking a clean visualization or a fluent chain-of-thought for a faithful account; both are optimized to look convincing, which is exactly the property that should raise your guard. Third, assuming that because a probe decodes a concept the model must be using it — amnesic probing shows this is often false. Fourth, over-reading a single successful intervention: steering one feature proves that direction matters, not that you have understood the whole circuit it sits in.
What's still debated, and where to go next
Step back and the open questions are large. The hottest concerns the very tool that excited the field most. Do sparse autoencoders recover the model's own features, or do they impose a tidy sparse basis that the dictionary itself invented? Critics point to feature splitting (train a bigger dictionary and one feature fractures into many, suggesting the 'unit' is partly arbitrary), to large unexplained residual variance (sometimes called the 'dark matter' of the activations), and to a tension between reconstructing activations well and yielding interpretable features. Through 2024 the field began building more principled SAE evaluations precisely because reconstruction loss and human-rated interpretability are not the same thing, and a feature can score well on both while still not being how the model computes. Whether SAEs cut nature at its joints or at convenient ones is genuinely unsettled.
A second, structural worry is the streetlight effect: we study the behaviors and models we can study — small networks, isolated circuits, clean toy tasks — and it is unproven that these methods scale to a frontier model with hundreds of billions of parameters whose interesting behaviors are diffuse and entangled. This is the heart of the interpretability gap: capability is sprinting while reliable understanding is walking. Reverse-engineering one induction circuit is a real triumph; certifying that a deployed model harbors no hidden goal to deceive is a different order of problem, and the distance between them is large and only partly charted. Optimists argue the gap is closing fast with SAEs and automation; skeptics argue we are decorating the easy cases while the hard ones stay dark. Both are serious positions held by serious researchers.
Underneath sits the deepest disagreement, and it is honest to leave it open. Some researchers believe interpretability is our best hope for a real safety case — a positive argument that a system is safe because we can see that it is — and that behavioral testing can never be trusted against a capable deceiver. Others believe interpretability is too immature to bear that weight on any near-term timeline, and that we should lean on behavioral evaluations and oversight instead. There is even a dual-use worry: the same understanding that lets us audit a model could help make it more capable, or teach a model how to route around our tools. This guide does not adjudicate that; the responsible reading is that interpretability is a fast-moving, partly-validated research program with real wins and real unknowns — not a finished instrument you can simply point at a model and trust.
Where this leaves us, and what comes next. The honest summary of the whole rung is humbling: we can sometimes read a model's insides, the tools are improving quickly, and we have learned to distrust an interpretation until intervention earns it. When we cannot read the internals reliably — which is still most of the time — we fall back on watching what the model does: behavioral testing, red-teaming, and capability evaluations, the subject of the next rung. But that fallback carries its own version of today's problem. Just as a plausible interpretation can be unfaithful, a model can pass a safety eval on the tested distribution and still fail elsewhere, or even sandbag — deliberately underperform to look harmless. The throughline of AI safety holds to the end: the hard part is never the behavior you can see, but trusting what you cannot.