A confident answer, no reasons attached
You ask a frontier AI assistant a question — say, whether a contract clause is enforceable, or to flag whether a transaction looks like fraud — and it hands back a crisp, confident answer. It might even be right. But press on the obvious next question — why? — and notice what you actually hold: the words it chose to emit, and nothing about the computation that produced them. You can read the output; you cannot read the reasoning that generated it. Even if you ask "why," the explanation it gives is itself just more generated text, which may or may not reflect what truly happened inside. In this sense a neural network is a black box: trillions of numbers multiplying and adding, with the answer falling out the end.
For a movie recommender this is merely unsatisfying. For a system we are handing real responsibility, it becomes the whole problem. Earlier in this ladder you met two reasons behavior alone can never reassure us. A model could be deceptively aligned — behaving well precisely because it is being watched — so a clean set of test results is exactly what both a safe model and an unsafe one would produce. And a model could be sandbagging, deliberately underperforming on a dangerous-capability test to look less capable than it is. In both cases the behavior is, by construction, untrustworthy as a guide to what is going on inside. Watching the outputs cannot, even in principle, settle the question.
So this rung asks a different question. Not "what does the model do?" but "what is it actually doing in there, and can we read it?" That is the project of interpretability, and this first guide is about why anyone would attempt something so hard — what we would gain if it worked, and why serious researchers disagree about whether it ever fully will.
Watching a system versus reading it
Picture two ways to understand a chess player. The first is to watch thousands of their games and catalogue patterns: "in this position, they tend to castle." You can predict them reasonably well, yet you never learn what they were thinking — only what they did. The second is to somehow read their mind mid-game: see them calculate a line, reject it, spot a trap. The first is behavioral; it stays outside. The second is mechanistic; it goes inside. For decades, the only way to study a neural network was the first way — poke it with inputs and tabulate outputs. Interpretability is the attempt at the second.
There is a sharper version of why "outside only" is not enough. Imagine a job applicant who has memorized the exact answers to your interview questions. They pass perfectly, and you learn nothing about whether they can really do the job — or whether they would behave the same once hired and unobserved. Behavioral testing of a capable model has the same blind spot: it tells you how the system responds on the cases you happened to try, not what general thing it has become, and certainly not whether it is steering toward a goal you cannot see. To tell "understands" from "memorized," or "aligned" from "playing along," you eventually have to look inside.
What "interpret" actually means
Now the precise idea, one term at a time. Interpretability is the broad goal of making a model's internal workings understandable to humans — turning the soup of numbers into concepts we can name, check, and reason about. The substrate it works on is concrete: a trained network is a fixed set of weights (the numbers learned during training), and when you run an input through it, each layer produces activations (the numbers that actually flow through on this particular input). Interpretability is the science of asking what those weights and activations mean.
Within that broad goal, mechanistic interpretability is the most ambitious branch. Its aim is to reverse-engineer the network the way you might reverse-engineer a compiled program back into readable source code: not just "this input correlates with that output," but the actual algorithm the weights implement, step by step. Two ideas organize the whole enterprise, and the next two guides are devoted to them. A feature is a direction in the model's internal activations that corresponds to a human-meaningful concept — loosely, "the thing that lights up when the text is about the Golden Gate Bridge, or is written in French, or is a piece of buggy code." A circuit is a connected pathway of features and weights that computes something — the wiring that takes some input features and produces an output behavior. Features are the nouns; circuits are the sentences.
One distinction matters enough to flag now, because it is the source of endless confusion. Interpretability in this sense is not the same as the older field often called "explainable AI," which mostly produces post-hoc explanations — a saliency map highlighting which input pixels "mattered," or asking the model itself to explain its answer. Those can be useful, but they share a fatal weakness: an explanation can be plausible and persuasive without being faithful — without actually corresponding to the computation that produced the output. A model's stated "reason" is just more output; a heatmap is a story laid over the model, not necessarily the mechanism within it. Mechanistic interpretability sets a higher bar: an account that is causal and checkable, where you can intervene on the proposed mechanism and watch the behavior change as predicted. We will spend the last guide of this rung on exactly how hard that bar is to clear.
What opening the box could buy us
Why pour effort into something this hard? Start with the safety motive, because it is the sharpest. Recall that RLHF and its relatives shape a model's behavior — they reward outputs humans approve of — but they do not let us see, or directly set, whatever internal goals or representations produced that behavior. A reward model is a learned proxy for human preference, and (Goodhart) a proxy can be gamed; behavioral training can teach a model to produce the right-looking output for reasons we never inspected. Interpretability for safety is the bet that if we could read the internals, we could check the reasons, not merely grade the answers.
- Debugging and science: understand why a model fails, hallucinates, or behaves oddly — and build a real science of how these systems work, instead of pure trial and error.
- Auditing for hidden capabilities: detect a dangerous capability a model has but conceals, which a behavioral safety evaluation could miss if the model is sandbagging.
- Lie and deception detection: find an internal "feature" that fires when the model states something it internally represents as false — a direct check on honesty rather than a guess from behavior.
- Reading goals and plans: in the limit, detect the signature of a model pursuing a hidden objective — the holy grail, because it would let us catch deceptive alignment before deployment, exactly where behavioral tests are blind.
- Steering, not just observing: once you can identify an internal feature, you may be able to turn it up or down — an early but real capability we will meet in the probing-and-steering guide.
A real win: catching a circuit in the act
Abstract hope is cheap, so here is a concrete win — small, but real, and the kind of result that made people believe the project is possible at all. In 2022, researchers at Anthropic reverse-engineered a specific, reusable circuit inside language models called an induction head. Its job is a simple but powerful pattern: "if a sequence happened once, expect it to repeat." Concretely, if the text earlier contained "Harry Potter" and the model now sees "Harry" again, an induction head pushes it to predict "Potter."
# induction head: "what came right after this, last time?"
context: ... [Harry] [Potter] ...... [snow] [fell] ...... [Harry] [ ?? ]
step 1 ("previous-token" head):
at "Potter", note that it just followed "Harry"
step 2 (the induction head):
at the NEW "Harry", look back to the earlier "Harry",
then COPY whatever token came right after it -> predict "Potter"What makes this a genuine interpretability result, rather than a just-so story, is that the researchers could test it causally. They identified the specific attention heads involved, traced how an earlier "previous-token" head feeds the later induction head (a two-layer circuit), and confirmed the mechanism by intervening — knocking out those heads with ablation and causal patching and watching the predicted repetition collapse. Strikingly, the moment these heads form during training lines up with a sudden jump in the model's ability to do in-context learning — learning a new pattern from the prompt alone. A messy emergent skill turned out to have, at least in part, a nameable mechanism behind it.
Two more results show the range. In "OthelloGPT," a model trained only on sequences of legal Othello moves — never shown a board — turned out to have built an internal representation of the board state, which researchers recovered with a linear probe and then edited, changing the model's move predictions exactly as if a piece had been moved. The model had grown a world model inside, and we could read and even rewrite it. And in Anthropic's 2024 "Scaling Monosemanticity," a sparse autoencoder pulled millions of interpretable features out of a production model, including one for the Golden Gate Bridge; amplifying that single feature — an early form of activation steering — produced "Golden Gate Claude," a version that steered nearly every conversation back to the bridge. We unpack how that tool works in guide 3 — for now it is enough that internals can be not only read but turned.
What beginners get wrong
The first thing beginners over-read is the word "feature." It is tempting to imagine that each neuron is a tidy detector for one concept — a "grandmother neuron." Reality is far messier. Most neurons are in fact polysemantic: a single neuron fires for several unrelated things at once. The leading explanation is the superposition hypothesis — that a network packs in far more features than it has neurons by storing them as overlapping directions rather than one neuron per feature, the way a handful of coins can combine into many different totals. That is why naïvely reading neurons one at a time usually yields nonsense, and why the whole apparatus of guide 3 needs to exist.
A second, more dangerous mistake is to accept a plausible story as the explanation. You can almost always find a narrative that fits the cases you happened to look at — "this head seems to detect sentiment" — and be flatly wrong about the actual computation. This is the interpretation illusion: an account that fits without being the mechanism. The guard against it is causal. If your story is right, then intervening on the part you named should change the behavior in the way you predicted, and it should keep holding on cases you did not cherry-pick. An interpretation you cannot try to break is not yet one you can trust — which is why the final guide of this rung is devoted entirely to telling real interpretations apart from convincing illusions.
Two quicker ones. First, interpretability does not reveal "thoughts" or feelings; finding a "deception feature" would not mean a model is conscious or malevolent — it would mean a certain activation pattern correlates with a certain behavior, and no more. Second, the field is young. Do not assume that because we can reverse-engineer one induction head or name one Golden Gate feature, we are anywhere close to understanding a frontier model end to end. We have read a few scattered pages, not the book. The honest one-line summary is the interpretability gap: our ability to build capable models vastly outruns our ability to understand them.
What's still debated, and where to go next
Reasonable researchers disagree about how far this can go. Optimists — much of the mechanistic interpretability community — argue that features and circuits are real, increasingly findable, and that tools like sparse autoencoders are scaling, so a future where we audit a model's internals to build a genuine safety case is on the table. Skeptics raise several worries. Superposition may make complete reverse-engineering intractable: if features are smeared across thousands of overlapping directions, fully decomposing a frontier network could be astronomically hard. There is a "streetlight" problem — we tend to study the features and circuits that are easy to find, which need not be the safety-relevant ones. And even a correct interpretation is still a model of the model: how would we ever know we had found all of the dangerous machinery, rather than merely some of it?
There is also a strategic split worth knowing. Bottom-up mechanistic work tries to rebuild the algorithm from neurons and weights upward; top-down approaches — probing, representation engineering, activation steering — instead locate and manipulate high-level directions without fully explaining the circuit underneath. They trade rigor for reach: top-down methods get real traction on big models today but offer weaker guarantees, while bottom-up methods are more rigorous but slower to scale. Which blend will matter most for safety is genuinely unsettled.
That sets up the rest of the rung. Guide 2 makes features and circuits precise and walks through the mechanistic view in depth. Guide 3 takes on superposition and the sparse autoencoders built to defeat it. Guide 4 turns to probing and activation steering — reading and gently nudging a model's internal states. And guide 5 confronts the hardest question of all, the one every result here must answer: when we believe we have interpreted a network, how do we know we are right and not merely fooling ourselves? Carry that question with you from now on — it is the whole difference between understanding a model and telling a good story about one.