the superposition hypothesis

Suppose you have ten light switches but a thousand things you want to signal. You could agree on a code where each message is a particular combination of switches, and as long as you rarely send two messages at once, you can pack far more meanings than you have switches. The superposition hypothesis says neural networks do something like this: they represent many more features than they have neurons, by storing features as overlapping combinations of directions rather than one neuron per feature.

The reasoning is geometric. A layer with n neurons has only n perpendicular directions, so it could store at most n features cleanly separated. But the world has far more useful concepts than that. Networks appear to exploit the fact that in high-dimensional space you can pack many almost-perpendicular directions, and that most features are rare (off most of the time). By tolerating a little interference, the model squeezes thousands of features into a space that 'should' only hold a few hundred — trading a small amount of noise for a big gain in capacity.

Superposition is, as the name says, a hypothesis — well-supported by toy models and increasingly by real ones, but still a working theory, not a proven law. It matters enormously for interpretability and therefore for safety: if features are squashed together and smeared across neurons, then looking at one neuron tells you little, and the clean concepts we want to audit are hidden in the overlaps. That diagnosis is the main reason tools like sparse autoencoders were invented to try to unpack the superposition.