probing LLM representations

Probing asks a blunt question: is some piece of information present in the model's internal vectors? You freeze the model, collect its hidden activations on many inputs, and train a small, separate classifier — a probe — to predict a property of interest from those activations, such as part of speech, sentiment, the truth of a statement, or even the board state implied by a game transcript. If the probe succeeds, the information is linearly readable from the representation.

It is popular because it is easy and revealing. A simple linear probe that recovers a property well suggests the model encodes that property in an accessible, roughly linear form; a probe that fails suggests the information is absent, tangled, or stored nonlinearly. Probing across layers shows where in the stack different kinds of information become available, painting a developmental picture of the representation.

But probing measures availability, not use. A property being decodable does not mean the model relies on it to produce outputs — a strong probe can even learn the property itself if you let it be too powerful. This is why probing is best paired with causal tools like activation patching, which check whether the encoded information is actually doing work downstream.