LLM interpretability

A trained language model is a giant pile of numbers that somehow turns words into more words. Interpretability is the effort to open that box and answer a simple-sounding question: what is the model actually doing in there? Instead of treating the network as a black box judged only by its outputs, we try to read the computations happening between input and answer — the way a biologist dissects a body to see which organ does what.

In practice this spans a ladder of ambition. The lightest forms just correlate inputs with outputs or highlight which words mattered. Deeper forms reach inside, decoding the model's internal vectors into human concepts and tracing how those concepts get combined. The goal is not a vague story but a causal account you could test: if this part computes that, then disabling it should change the behavior in a predictable way.

Why bother? Because we deploy these models in high-stakes places without knowing why they answer as they do. Interpretability is how we hope to catch hidden failure modes, verify a model is reasoning rather than memorizing, and eventually trust — or distrust — what it tells us. It is young, partial, and humbling: we can explain toy circuits cleanly but still understand only fragments of a frontier model.