concept and knowledge localization

When a model knows that the Eiffel Tower is in Paris, where exactly does that fact live? Knowledge localization tries to pin down which weights and which layers store a specific piece of information, rather than treating knowledge as smeared evenly across the whole network. The hope is that facts are modular enough to find — and, once found, to inspect, edit, or correct without retraining the entire model.

The main tool is causal tracing: corrupt the subject token, then patch clean activations back layer by layer to see which restorations revive the fact. This kind of analysis suggested that many factual associations are concentrated in the middle-layer feed-forward blocks, which behave like a key-value memory. Methods like ROME and MEMIT build on this to edit a fact by directly rewriting the implicated weights.

The picture is more tangled than early results implied. Localizing where editing succeeds is not the same as finding where a fact is stored — facts are often distributed and redundant, edits can have side effects on related knowledge, and a model may retrieve the same fact through several routes. Localization is a real signal, but a partial and contested one.