neuron interpretation

The most basic interpretability move is to pick one neuron and ask what makes it fire. You scan a large corpus, collect the inputs that drive a neuron's activation highest, and look for a pattern: maybe it spikes on French text, on the word "not," on the inside of quotation marks, or after a number. A neuron with a crisp, consistent trigger is monosemantic and gives a tidy little label.

At scale this has been automated: one model writes a natural-language guess for what each neuron of another model detects, then that explanation is scored by predicting the neuron's activations on held-out text. It is a way to put a first-pass caption on millions of neurons. The exercise also reveals structure — neurons for syntax, for specific tokens, for abstract features — distributed across the layers.

The hard truth is that most neurons are polysemantic: a single neuron fires for several unrelated concepts, because superposition packs many features into few neurons. So neuron labels are often only partial, and the field has largely moved toward sparse autoencoder features, which carve activations into cleaner, more monosemantic units than raw neurons provide.