a linear probe

Suppose you suspect a model secretly keeps track of something — say, whether a sentence is in the past tense — even though you never asked it to. How would you check what information is sitting in its internal activity? A linear probe is the simplest test: you take the model's activations and train a tiny, simple classifier on top of them to predict the property. If that simple classifier succeeds, the information must have been present in the activations in an easy-to-read form.

The 'linear' part is the whole point and the discipline. The probe is only allowed to draw a single straight dividing line (technically, read off one direction in activation space), so it cannot do clever computation of its own. That keeps the credit where it belongs: if a mere linear readout can recover 'is this text toxic?' from a layer's activations with high accuracy, then the model has already computed and stored that concept in a clean, linearly-accessible way. You freeze the model, collect its activations on labelled examples, fit the probe, and measure how well it predicts the label.

Probes are a cheap, popular first look inside a model and a building block for safety ideas like detecting whether a model 'knows' an answer it is not saying. But two honest cautions apply. First, a probe shows correlation — that the information is present and readable — not that the model actually uses it to decide its output; for that you need causal tests like activation patching. Second, a powerful probe can find structure that the model never relies on, or even latch onto patterns in your data rather than in the model, which is one classic route to an interpretation illusion.