faithfulness of explanations

An explanation can be convincing and still be false. Faithfulness asks whether an explanation actually reflects the computation the model performed, as opposed to merely sounding reasonable to a human. The distinction is sharp for language models because they can narrate their own reasoning — a chain of thought — that reads beautifully yet may have nothing to do with how the answer was really produced. Plausible is not the same as true.

The danger is everywhere in interpretability. A clean attention map, a tidy neuron label, a confident self-explanation, or a neat circuit diagram can all be post-hoc rationalizations. Models have been shown to give reasoning that ignores the cue that actually swayed them, and to produce explanations optimized to satisfy a reader rather than to disclose the mechanism. An unfaithful explanation is worse than none, because it breeds false confidence.

This is why the field insists on causal tests. The way to earn faithfulness is intervention: ablate, patch, or steer the components an explanation names and check the behavior moves exactly as predicted. Faithfulness is the quality-control standard that separates interpretability as evidence from interpretability as storytelling, and it is the bar every method here ultimately has to clear.