The black box problem
A large language model is a stack of matrices with billions of numbers in it. We trained those numbers by showing the model text and nudging them to predict the next token well — but nobody wrote the resulting behaviour. The model can translate poetry, debug code, and refuse a dangerous request, yet if you ask why it produced a particular answer, the honest default is: we don't fully know. That gap is the subject of LLM interpretability (可解釋性): the project of turning a trained network from an opaque artefact into something we can read, predict, and trust.
A multilayer network diagram: input, hidden, and output layers connected by many weighted edges.
Two kinds of explanation
It helps to split the field in two. Behavioural (or black-box) interpretability studies the model from the outside: feed it inputs, watch outputs, find patterns. It overlaps with classic explainable AI methods — saliency maps, feature attributions, asking the model to explain itself. Mechanistic interpretability instead opens the box and studies the internal computation: which components light up, what information flows through them, and how that produces the output. The LLM-specific version of this is mechanistic interpretability (機制可解釋性), and it is where most of this track lives.
A saliency map highlighting the input words that most influenced the model's output.
The dividing line matters because the two answer different questions. Behavioural methods tell you that the model relies on a word; mechanistic methods aim to tell you how a specific circuit computes with it. The grand ambition, borrowed from the wider mechanistic interpretability programme in machine learning, is to reverse-engineer a network into something like readable source code.
When an explanation is faithful
A pretty story about the model is worthless if it is wrong. The central quality bar in this field is faithfulness (忠實度): does the explanation actually describe the computation the model performs, or just one that sounds plausible? This is a sharp warning about an old habit — treating attention weights as explanations. A head attending strongly to a word looks like evidence that the word caused the output, but attention is correlational; the only way to know is to intervene and check.
Why it is worth the trouble
Three motives drive the field. First, debugging: if a model leaks training data, prefers one demographic, or hallucinates citations, you want to find the responsible mechanism rather than play whack-a-mole with prompts. Second, safety: interpretability is a load-bearing tool for LLM safety — detecting deception, auditing for hidden behaviours, and verifying a model is actually aligned rather than merely acting aligned during evaluation. Third, science: these are the most capable artefacts we have built, and understanding how they represent language and knowledge is interesting in its own right.
A map of the toolkit
The rest of this track climbs from what the model represents up to how it computes. We start with representations — the residual stream, features, and the surprising idea of superposition. Then we use sparse autoencoders to pull tangled features apart. Then we read attention directly and project hidden states to vocabulary with the logit lens. Finally we reach causal methods — patching activations and discovering circuits — which is where explanation graduates from a story into a tested claim.
- Guide 2 — representations: the residual stream, features, superposition, neurons, probing.
- Guide 3 — sparse autoencoders: pulling overlapping features into a clean dictionary, and steering with them.
- Guide 4 — attention: head roles, induction heads, and the logit lens.
- Guide 5 — causality: activation patching, circuit discovery, concept localization, steering, and faithfulness revisited.