Reading a mind, and nudging it
Suppose you want to know whether a chatbot "knows" it is being asked to do something harmful before it answers. You could just ask it, but a large language model can confidently say things that are not true about itself, so its self-report is weak evidence. There is, however, another place to look. As the model processes your prompt, real numbers flow through its layers, and somewhere in that flood the relevant information may sit in plain sight. This guide is about two practical ways to get at it: one that reads those internal numbers, and one that writes to them.
Here is the intuition before any jargon. Imagine a sealed machine with a bundle of wires inside. Probing is clipping a sensor onto one wire and learning to predict, from its voltage alone, whether a certain light will turn on. You never open the manual; you just learn to read the machine's state from the outside. Steering is the bolder next move: instead of only reading the wire, you inject a current into it and watch the machine's behavior shift. Reading tells you what information is present; writing tells you which information actually matters.
These two moves are the experimental backbone of mechanistic interpretability, and they connect directly to the ideas earlier in this rung. You have already met the picture of a feature as a direction in activation space, and the superposition-and-sparse-autoencoder story for fishing those directions out. Probing and steering are how you put that picture to the test: probing asks whether a feature can be read off the activations at all, and steering asks whether that direction actually drives the model. Think of them as the model interpreter's two basic gestures, look and push.
Probing: reading a concept off the activations
Now the precise idea, one term at a time. A linear probe is a very small, simple classifier — often just a single linear layer, a logistic regression — trained on the model's frozen internal activations to predict whether some property holds. Is this sentence about sports? Is this token a noun? Is the next move likely to be legal? You run the big model, freeze it, grab the hidden vector at some chosen layer, and train the tiny probe on top of those vectors. If the probe predicts the property well, we say the concept is linearly decodable at that layer. The technique was popularized by Alain and Bengio in 2016 precisely as a way to peer into intermediate layers.
A close cousin needs no training at all. The logit lens takes the model's own output layer — the matrix that normally turns the final hidden vector into next-token probabilities — and applies it early, to the hidden vector at an intermediate layer. The result is a guess at what the model "would say next" if it stopped thinking right there. Watched across layers, the logit lens shows a prediction sharpening up as it climbs the network, an oddly intimate view of a guess being assembled. It is, in effect, a fixed probe built from the model's own machinery.
Steering: writing a direction back in
If probing is reading, activation steering is writing. The idea is disarmingly simple: while the model is generating, add a chosen vector to the activations at some layer, and the model's behavior shifts in the direction of that vector. The classic recipe for building the vector uses contrast. Collect a set of examples that have a property — loving sentences, say — and a set that lack it, run the model on both, and take the difference of their mean activations. That difference is a candidate "concept direction." Scale it and add it in during generation, and you nudge the model toward the property, with no retraining and no gradient descent at all.
- Pick a concept and assemble contrast pairs: examples that have it (loving, or wedding-themed) against examples that do not (hateful, or neutral).
- Run the model on both sets and collect the hidden activations at a chosen layer.
- Compute the steering vector as the mean of the positive activations minus the mean of the negative ones.
- During generation, add alpha times the steering vector to that layer's activations on every token.
- Sweep the strength alpha: too small does nothing, too large breaks fluency into babble; you want the band where behavior shifts but the text stays coherent.
This is not a thought experiment. In the Activation Addition work (Turner and colleagues, 2023), adding a "Love" minus "Hate" vector to a small GPT made it veer toward affectionate, wedding-flavored text without any retraining. The broader research program of representation engineering (Zou and colleagues, 2023) takes this top-down: it extracts reading vectors for concepts like honesty, harmlessness, and emotion, then turns them into control knobs. The appeal for safety is obvious — steering is cheap, fast, and reversible, so you can imagine dialing a "toxicity" direction down at inference time without touching the weights. Whether that appeal holds up is a question we return to below.
The causal bridge: intervention, patching, ablation
Recall the caveat: a probe is correlational. It tells you a concept can be read, not that the model uses it. The honest way across that gap is causation — change the activation and see if the output changes the way your theory predicts. The workhorse here is activation patching, sometimes called causal intervention. Run the model on a clean input and save its activations; run it on a corrupted input that gives a different answer; then copy a single clean activation into the corrupted run at one specific spot. If the answer flips back, that spot carries the information that matters. Its blunter sibling is ablation: zero out or average-out a component and see what breaks.
# PROBING -- can a concept be READ off the activations? (correlational) acts = model.activations(prompts, layer=L) # [N, d] hidden vectors probe = train_linear_classifier(acts, labels) # is the concept present? score = probe.accuracy(heldout_acts, heldout_labels) # high score -> concept is LINEARLY DECODABLE at layer L # but decodable != used: you have shown correlation, not causation # STEERING -- WRITE a direction back in and watch behavior change (causal) v = acts[label==1].mean(0) - acts[label==0].mean(0) # candidate direction model.add_to_activations(layer=L, vector=alpha * v) # push along v # if outputs shift as predicted -> v is causally linked to behavior, # not merely present. reading + intervention is the strong combination.
Notice what this means for steering. Adding a direction and watching behavior change in the predicted way is itself a causal intervention. So steering does double duty: it is a control knob, and it is a test of a probe's finding. This is why the most convincing interpretability results pair the two gestures — a probe to show a concept is present (reading), and an intervention to show the model leans on it (writing). Correlation plus causation is a far stronger claim than either alone, and keeping that pairing in mind is the best defense against fooling yourself.
Worked examples: a board, a bridge, and a refusal
The cleanest demonstration of reading-plus-causation is Othello-GPT. Li and colleagues (2023) trained a GPT only on sequences of Othello moves — no board, no rules, just move tokens. A probe then recovered the current board state from the model's activations, evidence that the network had built an internal "world model" of the game it was never shown. Crucially, they did not stop at reading. Intervening on the probe-identified board representation changed which moves the model predicted as legal, showing the board was genuinely used, not a passenger. A follow-up by Neel Nanda sharpened the lesson: the board is cleanly linear only when you frame each square as "mine versus the opponent's" relative to the current player, not as a fixed black-versus-white — a vivid reminder that what you probe for shapes what you find.
A second example links steering straight back to the sparse-autoencoder story from the previous guide. In 2024 Anthropic scaled a sparse autoencoder on a production model, Claude 3 Sonnet, and among millions of features found one that fires on the Golden Gate Bridge. When they clamped that single feature to a high value, the result — nicknamed "Golden Gate Claude" — brought the bridge into nearly every answer, at one point describing itself as the bridge. That is steering at the level of a named feature: the autoencoder did the reading, clamping did the writing, in one public, concrete demonstration of feature-level control.
The third example is the most safety-relevant. Arditi and colleagues (2024) reported that across many open chat models, refusal — the behavior of declining a harmful request — is mediated largely by a single direction in activation space. Ablate that direction and the model stops refusing, a jailbreak performed by editing activations rather than crafting a clever prompt; add the direction and the model refuses even harmless questions. This cuts both ways and is worth stating plainly: steering is a potential safety tool (turn unwanted behavior down) and a potential attack surface (turn safety off). And note the scope honestly — this works where you can touch the activations, as with open-weight models, and it is strong evidence about those models, not a proven law of all of them.
Common misconceptions and pitfalls
The first and biggest pitfall we have already named: "the probe found the concept, so the model uses it." Reading is not using, and the only cure is intervention. Worse, a high probe score can come from the probe itself being too powerful — a sufficiently flexible classifier can fit almost anything, including random labels, so accuracy alone proves little. The standard guard, from Hewitt and Liang's work on control tasks, is to also train the probe on a meaningless control version of the task: if it scores nearly as high on nonsense, your probe is memorizing, not reading the representation. Restrict to simple linear probes and check this selectivity before believing a result.
The second pitfall mirrors it: "steering worked, so I found the true feature." A steering success shows the direction is causally relevant to the output, which is real and useful — but the direction may be an entangled mush rather than a clean concept. Remember the superposition hypothesis: features can be packed together so a single direction is polysemantic, and pushing it moves several things at once. The effect can also route through an unexpected path, or a large strength can simply break the model into babbling about your topic by damaging it, not by cleanly invoking a concept. A clean demo is evidence, not proof of mechanism.
The third pitfall is treating either tool as understanding itself. Probing and steering are powerful but coarse instruments. A probe tells you what is linearly decodable by your probe; a steering vector tells you a direction nudges behavior. Neither hands you the model's algorithm — the step-by-step computation that actually produces the answer. They are how you generate and test hypotheses about that algorithm, not the explanation. That is exactly the worry the next guide takes up under the heading of the interpretation illusion: a story that fits the evidence without being the real mechanism.
What is still debated
The first live debate is the epistemics of probing itself, an argument that has run in the language-processing community for years. How much does a probe tell us about the model, versus about the probe? Beyond Hewitt and Liang's control tasks, researchers have proposed information-theoretic framings (Voita and Titov; Pimentel and colleagues) that recast probing as measuring how much a concept is "extractable" rather than whether the model encodes it. There is no settled methodology, and "a probe got high accuracy" remains contested as evidence about what a network really represents.
A second debate sits underneath both tools: the linear representation hypothesis. Probing and steering largely assume concepts live as approximately straight-line directions you can read and add. This is strongly supported empirically — an enormous amount works as if it were true — but it is not known to be universal. Recent work (for example Engels and colleagues, 2024) has found genuinely multi-dimensional, even circular, representations, such as the way days of the week or months can sit on a loop. If important features are nonlinear, a linear probe will distort them and a single steering vector will miss them. How far linearity carries us is open.
A third debate is whether steering is reliable enough to count as a safety intervention rather than a clever parlor trick. Steering vectors can be brittle, sensitive to the layer, the strength, and the exact prompt, and pushing them often costs some general capability. Whether "edit the activations to make it safe" is robust enough to deploy is unresolved. The same hope, used the other way, is whether a probe could serve as a kind of lie detector — a route toward eliciting latent knowledge, reading a model's hidden beliefs. It is genuinely promising and being actively studied.
Key takeaways and where to go next
Hold on to the pair. Probing reads a concept off the activations and is correlational; activation steering and other interventions write to the activations and are causal. Reading alone can mislead, writing alone can be a fluke, and the strongest interpretability work combines them — show the concept is present, then show the model leans on it. That look-and-push discipline is the most transferable habit you can take from this guide, and it is what makes interpretability a tool for safety rather than a source of comforting stories.
That last phrase sets up the final guide of this rung. Once a probe lights up or a steering vector works, a hard question remains: how do we know we have found the mechanism and not just a convenient story that happens to fit? Guide 5 takes that doubt head on, examining the interpretation illusion and asking, with real methods like causal scrubbing in mind, when an interpretation can actually be trusted. You now have the hands; next you sharpen the judgment that keeps them honest.