attention pattern visualization

Because attention weights are just numbers between zero and one, we can draw them. A common picture is a heatmap or a web of lines connecting each token to the tokens it attends to, with brighter or thicker links for stronger weights. Looking at these for a real sentence, you can often see a head that always points to the previous word, or one that links each pronoun to the noun it refers to.

These visualizations are a first window into a model's inner workings. By picking a layer and a head and plotting its n-by-n weight grid, researchers spot recurring motifs — diagonal stripes for local heads, vertical bands for tokens everyone attends to, neat arcs for matching brackets. Tools that render these patterns interactively helped reveal induction heads and other reusable circuits.

But a pretty pattern is not an explanation. High attention to a token does not prove that token caused the output, because the value being carried and later layers both matter, and softmax forces some weight to land somewhere. Attention maps are a valuable clue and a poor proof; rigorous interpretability pairs them with interventions that actually change the computation.