The neuron that means seven things
In the previous guide you learned to read a network as features and circuits, with the inviting dream that we might one day open a model the way an engineer opens a radio: trace the wires, name the parts, and see how it works. Then you actually go looking inside a real model — and the dream stumbles on the very first part you pick up. You select one neuron, expecting it to stand for one clean thing, and instead it lights up for an absurd grab-bag: photos of cats, the token "December", strings of DNA, and the syntax of a Python error message. It is not broken; this is the norm. Most individual neurons in a large model are like this, and that single observation is the puzzle this whole guide exists to solve.
This messiness has a name: a neuron that responds to several unrelated things is called a polysemantic neuron, and the property is polysemanticity. Why should you care? Because the whole promise of mechanistic interpretability — and the safety reason for it from the first guide in this rung — was to be able to point at a part of a model and say what it does. If we want to audit a large language model for a hidden deception feature, or check whether a refusal is real or a fragile surface trick, we need parts we can name. A part that means seven things at once is a part we cannot honestly name, and a model built almost entirely out of such parts looks, at first, like a dead end for understanding.
More ideas than there are wires
Before any jargon, here is the intuition. A given layer of a neural network has a fixed, finite number of neurons — say a few thousand. But a model that genuinely understands language and the world needs to keep track of an astronomically larger number of distinct concepts: every famous person, every programming language, every emotion, idiom, chemical, and grammatical mood. There are simply far more things worth representing than there are neurons to represent them. So the network faces a storage problem it cannot solve the easy way, with one neuron per concept. It has more ideas than it has wires.
Picture a tiny studio apartment that somehow has to hold the belongings of a large house. You cannot give every object its own shelf — there aren't enough shelves. But you can survive if you exploit one fact about how you actually use your things: you almost never need all of them at once. Most of the time only a handful are in play. So you stack and overlap, packing far more away than there are shelves, and on any given day you pull out just the few you need. The crucial enabling fact is sparsity: of all the concepts a model could represent, only a tiny fraction are actually active in any single input. The word "December" and a strand of DNA rarely matter in the same sentence.
So the network strikes a bargain. Because concepts are sparse, it can afford to overlay many of them in the same small space, accepting that just occasionally two will collide and interfere a little — a small, tolerable amount of noise in exchange for representing vastly more than it otherwise could. That bargain is the heart of everything that follows. The polysemantic neuron you met in the last section is not a bug; it is the visible fingerprint of a network that has chosen to pack more concepts than it has neurons. Now we make that precise.
Features are directions, and they overlap on purpose
The precise idea rests on one more concept from the previous guide, now stated sharply: the linear representation hypothesis. It says that a feature is represented not by a single neuron but by a direction in the high-dimensional space of a layer's activations — much like a concept can be a direction in an embedding space. A feature is "present" in an input to the degree the activation points along its direction. This reframing is liberating: directions are not limited to lining up with the neuron axes. A feature can point diagonally across many neurons at once, which is exactly why reading one neuron at a time told us so little.
Now the hypothesis itself. In a space with d dimensions you can fit at most d directions that are mutually perpendicular (orthogonal). If a network only ever used orthogonal directions, it could store exactly d features in a d-neuron layer — the easy one-per-wire scheme. The superposition hypothesis says networks do far better than that: they pack in many more features than dimensions by using directions that are merely nearly orthogonal. Almost-perpendicular directions interfere only slightly, and because features are sparse, the rare collisions rarely both fire hard at once, so the interference stays small enough to be worth it. The network is, in effect, simulating a much larger layer with imaginary extra neurons, paying for it in a little noise.
This closes the loop on the puzzle we opened with. If features are directions that do not line up with the neuron axes, then any single neuron sits at some angle to a whole crowd of feature-directions, and so it will fire — partly — whenever any of them is active. That is precisely why one neuron responds to cats and December and DNA at once: it is catching the spillover of several different features that happen to have a component along its axis. Polysemanticity, in other words, is not a separate mystery; it is the surface symptom, and superposition is the underlying cause. Cause and symptom, finally connected.
The toy model that made it concrete
Superposition could be just a tidy story, so it matters that there is a clean, controlled demonstration of it. In "Toy Models of Superposition" (Anthropic, 2022, Elhage and colleagues), researchers built a deliberately tiny network and handed it a task with a known answer: take a handful of features that are sparse and roughly independent, squeeze them through a layer with fewer dimensions than there are features, and reconstruct them. There is no room to store each feature on its own axis. What does the network do? When the features are dense (usually several active at once), it gives up and only stores the few most important ones orthogonally — the safe, boring solution. But as the features get sparser, the network flips into superposition, cramming in the extra features as non-orthogonal directions and tolerating the resulting interference.
The lovely part is that the network does not pack the directions in haphazardly. It arranges them into neat geometric figures — antipodal pairs, triangles, pentagons, and higher-dimensional shapes — that spread the directions out to keep interference as low as possible, the same way you would space points evenly around a circle. The switch from "store a few cleanly" to "superpose many" happens as a sharp phase change governed by how sparse the features are. None of this proves that a frontier model does exactly the same thing — a toy is a toy. But it converts superposition from a hand-wave into a phenomenon with rules you can derive, predict, and watch happen on demand. That is what made the rest of the field take it seriously.
Sparse autoencoders: pulling the chord apart into notes
If superposition is the network folding many sparse features down into few crowded dimensions, the obvious move is to run the fold in reverse: learn to unfold the crowded activation back into a wide, sparse list of features. Think of a layer's activation as a musical chord — several notes sounding at once, blended into one waveform — and the goal is to recover which individual notes are playing. The mathematical name for this move is dictionary learning: explain each activation vector as a sparse combination drawn from a large "dictionary" of basic directions, where only a few dictionary entries are switched on for any given input.
The tool that does this for neural networks is the sparse autoencoder, or SAE, and it is worth meeting one term at a time. An SAE has two halves. The encoder takes an activation vector from the model and projects it up into a much wider hidden layer — often eight to thirty times wider than the original, which is why it is called overcomplete. A nonlinearity plus a penalty force that wide layer to be mostly zeros, so only a few of its units switch on for any input. The decoder then takes those few active units and adds their directions back together to reconstruct the original activation as closely as it can. Training pushes on two things at once: reconstruct the activation accurately, and keep the active-unit count small.
# a sparse autoencoder over one layer's activations x (dim d_model)
# dictionary size d_sae >> d_model (overcomplete: many more units)
f = ReLU(W_enc @ (x - b_dec) + b_enc) # sparse feature code, size d_sae
x_hat = W_dec @ f + b_dec # reconstruct x from active features
loss = || x - x_hat ||^2 # (1) reconstruction: explain the activation
+ lambda * || f ||_1 # (2) sparsity: keep very few features on
# the COLUMNS of W_dec are the learned feature directions = the "dictionary"
# the nonzero entries of f say which features are present, and how strongly
# raise lambda -> sparser, more monosemantic, but worse reconstruction- Run the model over a huge corpus and record the activation vectors at one chosen site — for example, the residual stream at one layer.
- Train an overcomplete autoencoder on those vectors with a sparsity penalty, so it must explain each one using only a few active units.
- Read each decoder column as a candidate feature, and label it by collecting the inputs that make that unit fire most strongly.
- Check the label is real, not wishful, by intervening: clamp the feature on or off (steering or ablation) and see whether the output shifts the way the label predicts.
- Iterate — tune the dictionary size and sparsity, watch for dead features and feature splitting, and validate against held-out behavior.
What SAEs found, and Golden Gate Claude
Does any of this actually work on real models? The first convincing yes came from "Towards Monosemanticity" (Anthropic, 2023, Bricken and colleagues). They trained an SAE on the activations of a small one-layer transformer and pulled out thousands of features that were strikingly clean — monosemantic, meaning each one stood for a single recognizable thing. There was a feature for Arabic script, one for DNA sequences, one for base64-encoded text, one for legal language. They also noticed feature splitting: enlarge the dictionary and a single broad feature, say "mathematics", fractures into many finer ones — "matrix algebra", "topology", and so on — a hint that there is no single privileged list of features, only a resolution you choose.
Then came the scale-up. In "Scaling Monosemanticity" (Anthropic, 2024, Templeton and colleagues), SAEs were trained on Claude 3 Sonnet, a deployed production model, and recovered millions of features — including, famously, a feature that activates on the Golden Gate Bridge across text and images. The team also reported preliminary safety-relevant features: ones that track sycophancy, deception, dangerous code, and bias. Other labs reached similar ground by different routes — OpenAI trained very large top-k SAEs on GPT-4-class models, and Google DeepMind released Gemma Scope, an open suite of SAEs across a model's layers. The phenomenon is not one lab's artifact.
The most vivid proof that these features are not just labels stuck on after the fact came from intervening on them. By clamping the Golden Gate Bridge feature to a high value, Anthropic produced "Golden Gate Claude", a temporary public version of the model that steered nearly every conversation toward the bridge — asked for a cookie recipe, it would somehow route through San Francisco's fog and steel cables. That is activation steering, the subject of the very next guide: editing a model's behavior by turning a named direction up or down. When you can change behavior in the predicted way by pushing on a feature, you have evidence the feature is doing real causal work, not merely correlating with the output.
What beginners get wrong
The first and biggest mistake is to treat the features an SAE produces as the model's true, God-given atoms of thought. They are not. An SAE's output depends heavily on choices you made: the dictionary size and the sparsity penalty. Crank the dictionary larger and broad features split into finer ones; change the penalty and the whole inventory shifts. Two reasonable SAEs trained on the same model can disagree about what the features even are. So an SAE gives you a useful decomposition at a chosen resolution — a sensible carving of the activations — not a readout of the one true set of concepts the model "really" has.
A second mistake is to assume SAEs explain everything in the activation. They do not. There is always a reconstruction gap: splice an SAE's reconstruction back into the model in place of the real activation and the model gets measurably worse, because some information was lost. Researchers sometimes call the unexplained remainder the "dark matter" of SAEs, and it can be a substantial fraction. A related slip is to read a high reconstruction score as a high usefulness score — an SAE can reconstruct activations well while still missing the features that actually drive the behavior you care about.
A third mistake is to swallow the linear representation hypothesis as settled fact. It is a remarkably productive assumption, but not a proven law. In "Not All Language Model Features Are Linear" (2024, Engels and colleagues), researchers found features that are genuinely multi-dimensional rather than single directions — for instance, the days of the week and the months of the year are represented on small circles, so the model can do clock-like arithmetic such as "three days after Saturday." If some concepts live on curved or multi-dimensional surfaces, a method built to find straight directions will, by design, miss or distort them. "Features are directions" is a powerful first approximation, not the last word.
What's still debated, and where to go next
The sharpest open question is blunt: are SAEs actually useful for the things we want interpretability to do? After the initial wave of excitement, several 2024–2025 studies pushed back, finding that SAE-derived features did not reliably beat much simpler baselines on concrete downstream tasks — for example, probing for a specific concept or steering a specific behavior, where an ordinary linear probe sometimes did just as well or better. The worry is that the sparsity-and-reconstruction objective optimizes for explaining the activation, which is not the same as isolating the features that matter for behavior. Defenders reply that SAEs are young, that the evaluations are still immature, and that their unsupervised, open-ended discovery of features no one thought to look for is a real advantage a targeted probe cannot match. This is an active, healthy fight, not a settled verdict.
Underneath that sits a deeper uncertainty: even the superposition hypothesis is not fully nailed down for the largest models, and it is genuinely hard to know whether an SAE is measuring structure the model already has or imposing structure that merely looks tidy. This is one face of the broader interpretability gap — the distance between the confident stories we can tell and the rigorous causal understanding we actually want. The field is already moving in response: newer methods such as transcoders, crosscoders, and end-to-end training tweak the objective so it tracks the model's computation more directly, and Anthropic's 2025 circuit-tracing work leaned on cross-layer transcoders to map small reasoning steps end to end. The honest summary is that SAEs are among the most exciting interpretability tools of the last few years and are under heavy, deserved scrutiny — neither a solved problem nor a dead end.
Where to go next: the very next guide, on probing and activation steering, takes the features you now know how to find and asks how to test them and use them — turning a named direction up or down, reading a concept out with a linear probe, and the safety uses and limits of both. The guide after it, on evaluating interpretations, supplies the discipline this whole rung needs: how to tell whether a clean-looking feature is the real mechanism or just a flattering story. Carry forward the single idea that ties this guide to both: a feature you can name is a hypothesis about the model, and a hypothesis is only worth as much as the tests it survives.