Why read the wiring, not just watch the output
Picture two models sitting side by side. You give them the same thousand tricky prompts, and they answer identically — calm, safe, helpful, every single time. From the outside they are indistinguishable. But suppose one of them genuinely holds the values it displays, while the other is merely producing safe-looking answers because that is the smart move while it is being tested. The previous rung gave that worst case a name, deceptive alignment, and the opening guide of this rung drew the conclusion: behavior alone cannot tell these two apart, because passing the test is exactly what both would do. If watching the output has a ceiling, the natural next move is to stop watching and start looking inside.
This guide is the first map of what you would actually be reading once you open the box. The word interpretability covers a wide range of things — including asking a model to explain itself, or drawing a heat-map over which input words mattered most. Those describe behavior from the outside. This guide is about the more ambitious cousin, mechanistic interpretability, which treats a trained neural network not as a mysterious oracle but as a program — a real, fully determined computation — and tries to reverse-engineer the algorithm that program is running. To do that you need two pieces of vocabulary above all others: features and circuits. Get those, and the rest of this rung has somewhere to stand.
From black box to wiring diagram
Here is the cleanest analogy. Imagine someone hands you a compiled program — just the binary, billions of raw numbers — but not the source code. The numbers are the program: there is no hidden magic in there, only arithmetic, and the same input always gives the same output. The trouble is purely that this program is written in a form no human was ever meant to read. Reverse-engineering it — recovering a human-understandable account of what computation those numbers perform — is exactly what mechanistic interpretability attempts for the weights of a trained network. It is decompilation, applied to a program that gradient descent wrote and no one designed.
A second analogy helps with the spirit of it: this is like neuroscience, but with superpowers a brain scientist can only dream of. We can read every artificial "neuron" and every connection exactly, run the system forwards and backwards, freeze any intermediate value, and edit any piece to see what changes. The foundational vision for the whole field — laid out in the 2020 "Zoom In" paper by Chris Olah and colleagues — rests on a three-part bet about what we will find when we look. First, that the computation decomposes into features, the meaningful units. Second, that features are wired together by the weights into circuits. Third, that the same features and circuits tend to recur across different models — a claimed universality. That bet has held up impressively in some places and remains contested in others, which is the honest backdrop for everything below.
Why insist on the word "mechanistic"? Because it marks a specific ambition. You can describe a model from the outside ("it tends to refuse this kind of request"), or you can read a single internal signal (a probe — the subject of guide 4). The mechanistic goal is more than either: it is the full causal story of which internal pieces compute what, and how they connect to produce the final output. And crucially, a mechanistic claim is meant to be testable by intervention — you change a part and check the output changes in the way your story predicts, a method called causal intervention. That demand for causal evidence is the thread that runs all the way to guide 5, where we ask whether we can trust an interpretation at all.
What is a feature?
Start with the dream that got everyone excited, which came from biology. In 2005 neuroscientists recording from a single neuron in a human brain found one cell that fired for photographs of the actress Jennifer Aniston — and kept firing for drawings of her, and even for her name in text, but not for other people. It became famous as a "grandmother cell": the tantalizing idea that one neuron might cleanly encode one concept. The hope for artificial networks was the same — open one up and find the "dog neuron," the "is-this-French neuron." And sometimes it really is that clean: vision networks reliably grow neurons that detect edges, and neurons that detect curves. So here is a first definition. A feature is a property of the input that the network has learned to detect and represent internally.
Now sharpen it, because "a neuron that means X" is too crude. The activations of a layer are a long list of numbers — a vector — and the better way to think of a feature is as a direction in that activation space, not necessarily a single coordinate. "How much does the input have this property?" becomes "how far does the activation point along this particular direction?" The classic intuition comes from word embeddings, where the famous (if imperfect) example is that king − man + woman lands near queen: meaning like "royalty" or "gender" lives along directions you can add and subtract, not inside single numbers. The idea that features are directions like this is called the linear representation hypothesis — powerful, widely used, and, as we will see, not the whole story.
Here is the twist that broke the grandmother-cell dream. When researchers actually inspected real neurons, most of them turned out to be messy. A single neuron in a vision model might fire for cat faces and the fronts of cars and cat legs all at once — three unrelated things. That is a polysemantic neuron ("many meanings"), and it is the rule, not the exception. Why would training produce such a tangle? The leading explanation is the superposition hypothesis: a network wants to represent far more features than it has neurons, so it crams many features into the same neurons, storing them as a set of nearly-separable directions that overlap. This is precisely why you cannot just read meanings off raw neurons — and it is the reason tools like sparse autoencoders were invented to pull the overlapping features back apart. That is the whole subject of the very next guide, so we only plant the flag here.
What is a circuit?
If features are the nouns, circuits are the sentences. A circuit is a connected set of features wired together by the network's weights, which together implement some understandable sub-computation — a fragment of algorithm. The picture is concrete: features in earlier layers feed, through weights, into features in later layers, and a circuit is the particular subgraph of those connections that computes one specific thing. A real example from the vision work: a "car" detector is built by combining a wheel-detector feature found near the bottom of the image, a window-detector feature near the top, and a car-body feature, with the connecting weights effectively checking that these parts appear in the right spatial arrangement. That is not a metaphor for what the network might be doing — it is a description of wiring that was traced by hand.
In a language model the building blocks are not only the neurons inside the feed-forward layers but also attention heads — the pieces that move information between token positions. So a circuit in a transformer is typically a handful of attention heads and feed-forward neurons, spread across several layers, passing intermediate results from one to the next. The key mental shift is this: the unit of understanding is no longer "neuron number 4,217 means X." It is "these specific pieces, connected in this specific way, implement this specific algorithm." Heads compose — the output of one head becomes the input another head reads — and it is that composition, more than any single component, that does the interesting work.
- Pick a crisp, specific behavior to explain — for instance, "the model correctly completes a name it has already seen earlier in the text."
- Find which components light up when the model does it: which attention heads and neurons are active on those inputs?
- Form a hypothesis: this particular set of pieces, wired this particular way, is what computes the behavior.
- Test it causally — ablate or patch those components and check the behavior breaks, or transfers, exactly as your story predicts.
- Stay suspicious of a story that merely fits what you have seen; insist on the intervention evidence before you believe it.
A worked example: curve detectors and induction heads
Take vision first, where the circuits program began. The "Zoom In" team studied InceptionV1, an ordinary image classifier, and found curve detectors: neurons in an early-to-middle layer that fire for a curve at a particular orientation, and respond smoothly less as you rotate the curve away from that angle. They went further and traced where curve detectors come from — they are assembled out of simpler line and edge detectors in earlier layers — and how curves in turn feed into later detectors for things like dog heads and car parts. And they reported finding recognizably similar curve detectors in independently trained networks, which is the universality claim made concrete. This is the existence proof the whole field leans on: a real, hand-verified circuit you can inspect.
Now language, and the most celebrated circuit result in modern models: induction heads, from a 2022 Anthropic study. An induction head implements a beautifully simple copy-and-continue rule. If the current token is A, it looks back through the text for an earlier place where A appeared, sees which token came right after it — call that B — and predicts B again. So in a passage mentioning "Dr. Lee" early on, when "Dr." shows up again later, the head helps predict "Lee." It is, quite literally, pattern completion of the shape [A][B] ... [A] → [B], and it is one of the mechanisms by which a model can pick up and continue a pattern it has only just seen in its own context.
sequence: ... Dr. Lee ... went home ... Dr. ???
A B A
step 1: current token = Dr. (call this token A)
step 2: scan backwards for an earlier A -> found the first Dr.
step 3: read the token that followed it -> Lee (call this B)
step 4: predict B -> Lee
rule: [A][B] .......... [A] -> predict [B]The mechanistic beauty is in how it is built. An induction head does not work alone: it is formed by composing two heads across layers. A "previous-token head" in an earlier layer writes, into each position, a note saying "the token just before me was A"; the induction head in a later layer reads that note to locate where A occurred and copies the token that followed. Two simple pieces, wired together, compute a genuine capability. And there is a striking dynamical twist: as induction heads form during training, there is a visible bump — a phase change — that coincides with a sudden jump in the model's in-context learning ability. The evidence that induction heads are a major driver of in-context learning is strongest in small language models and more suggestive in large ones, which is exactly the kind of honest hedge this field should make.
One more, to show both the method's reach and its honesty: the IOI circuit. A 2022 study ("Interpretability in the Wild") reverse-engineered, in GPT-2 small, the circuit that completes sentences like "When John and Mary went to the store, John gave a drink to ___" with "Mary." The authors identified roughly two dozen attention heads working in functional groups — for instance "name mover heads" that copy the correct name to the output, and "S-inhibition heads" that suppress the already-mentioned subject so the other name wins — and they verified these roles by patching activations to confirm the behavior really depends on those components. It is the most detailed circuit yet recovered from a real language model, and the authors were candid that it is still incomplete and messy. That candor is itself an important data point about how hard full understanding remains.
What beginners get wrong
The first mistake is to equate a feature with a neuron. As we saw, most neurons are polysemantic; a feature is better understood as a direction that can be spread across many neurons, and a single neuron can carry pieces of many features. A second, very common mistake in language models is to think "interpretability just means looking at which words a head attends to." Attention patterns are suggestive, but they are not the mechanism. Knowing that a head attends from one token to another tells you nothing about what it then does with that information, and attention weights on their own can be actively misleading. The mechanism lives in the whole computation and has to be checked by intervening, not by eyeballing a pretty attention map.
The biggest mistake of all is to treat a clean story that fits the data as the same thing as the true mechanism. It is not. You can find a tidy narrative — "aha, this is the sentiment neuron" — that turns out to be an interpretation illusion: it correlates nicely on the examples you happened to look at, but it is not actually what drives the output, or it falls apart on new inputs. This is a documented failure, not a hypothetical one: a 2021 study showed that a single neuron in BERT could be made to look like it meant quite different things depending on which dataset you used to visualize it. That is exactly why serious mechanistic work insists on causal tests rather than suggestive pictures — and why guide 5 of this rung is devoted entirely to the question of whether an interpretation can be trusted.
What's still debated, and where to go next
The first live debate is whether any of this scales. The triumphs above are small models or narrow behaviors; frontier systems have hundreds of billions of parameters with superposition everywhere. Optimists point to real recent progress: in 2024 Anthropic used sparse autoencoders to extract millions of more-interpretable features from a production-scale model (Claude 3 Sonnet), including a now-famous "Golden Gate Bridge" feature that, when turned up, made the model bring the bridge into almost any conversation. Skeptics reply, fairly, that pulling out a big dictionary of features is not the same as understanding the whole computation — extraction is not yet end-to-end explanation — and that we remain far from a complete mechanistic account of any large model.
The second debate is whether the core hypothesis is even right. The linear representation hypothesis — features as directions — is enormously useful, but it is not universal: recent work has found genuinely multi-dimensional representations, such as days of the week arranged in a circle rather than along a line, so not everything is a clean additive direction. There is also the "streetlight" critique: we naturally study the behaviors and components we already know how to study, which may quietly bias the whole picture toward the parts that happen to be legible. And there is a methodological split between bottom-up circuits work (build understanding up from features) and top-down representation engineering, a 2023 approach that reads and steers high-level representations without first recovering a full circuit. Reasonable researchers weight these directions differently, and it is genuinely unsettled which will pay off most.
The third debate matters most for this ladder: does any of this actually buy safety yet? This is the interpretability gap — the distance between what we can interpret today and the reliable, scalable understanding we would need to do something like detect a deceptively aligned model by reading its goals straight off the weights. Most researchers are honest that we are not there. Mechanistic interpretability for safety is a promising bet, not a delivered solution, and some serious people argue that other approaches — stronger evaluations, or control techniques that work even without understanding — deserve as much or more of our effort. None of this is settled, and the responsible stance is to hold the disagreement openly rather than to pick a winner.
Where to go next. The very next guide opens up the superposition hypothesis and sparse autoencoders in depth — why neurons end up polysemantic, and how dictionary learning tries to recover clean, single-meaning features from the tangle. Guide 4 turns to lighter-weight tools that read or nudge representations without a full circuit: linear probes, activation steering, and the logit lens. And guide 5 confronts the question that has been hovering over this entire guide — how do you tell a real interpretation from a convincing illusion? Carry the three-question habit forward; you will need it for everything that follows.