Inside the Representation: Stream, Features, Superposition

The residual stream as a shared workspace

Every token in a transformer carries a vector that passes straight through the network, with each layer adding to it. That highway is the residual stream, and reading it is the foundation of residual stream analysis (殘差流分析). Think of it as a shared scratchpad: attention heads and feedforward blocks read from the stream, compute something, and write their result back by addition. Because contributions add up, you can in principle decompose any later hidden state into the sum of everything that wrote to it — a property mechanistic interpretability leans on heavily.

Features, not neurons

The unit of meaning we actually care about is a feature: a direction in activation space that corresponds to a human-recognisable concept — this text is in French, we are inside a quotation, the subject is a US president. The working hypothesis of the field is that the network's computation is best described as features and circuits (特徵與電路): features are the variables, and circuits are the algorithms that read some features and write others. Crucially, a feature need not be one neuron.

That is why interpreting individual neurons (神經元解釋) is frustrating. You probe one neuron and find it fires for DNA sequences and legal citations and HTML tags — a tangle with no clean story. These polysemantic neurons respond to several unrelated concepts, so reading them one at a time gives a misleading picture of the computation.

Superposition: more concepts than dimensions

Why are neurons polysemantic? The leading answer is the superposition hypothesis (疊加假說): a model wants to represent far more features than it has dimensions, so it packs them in as almost-orthogonal directions that share neurons. As long as only a few features are active at once — which is true for language, where most concepts are absent in any given token — the model can pull them apart again with acceptable interference. Superposition is a compression trick the network learns, and it is the single biggest reason raw activations look like noise.

Superposition packs many feature directions into one vector by keeping them nearly orthogonal, so each can be read back with little interference.

Probing: asking what a layer knows

Before we untangle features, we can at least test what information is present somewhere in the stream. Probing (探測) trains a small classifier — a probing classifier — to predict some property (part of speech, sentiment, whether a chess move is legal) from a frozen hidden state. If the probe succeeds, the information is linearly readable at that layer. It is a cheap, powerful way to map where and when a concept becomes available as the model processes a sequence.

A linear probe maps a layer's hidden state h to a label — high accuracy means the property is already encoded there.

# Probe layer L for 'is the sentence French?'
H = hidden_states(model, texts, layer=L)   # frozen activations
probe = LogisticRegression().fit(H, labels) # tiny linear classifier
print(probe.score(H_test, labels_test))     # high => info is linearly present at L