attention map
An attention map is a picture of where a transformer is looking. Inside every attention layer, each token assigns a weight to every other token — how much it draws from each one. For images these weights are spatial, so you can lay them back over the original picture as a heatmap: bright where a token attends strongly, dark where it ignores. The most common version asks, for the class token, how much attention it pays to each patch, and displays that as a saliency overlay revealing which regions the model relied on for its decision.
Formally, the attention weights are the rows of the softmax(QKᵀ/√d_k) matrix — for each query token, a probability distribution over all key tokens. Picking one query (often the class token), reshaping its N weights back onto the patch grid, and upsampling to image size gives one head's map; you can average over heads, inspect individual heads, or compare across layers. Because the matrix is computed at runtime per image, attention maps are a free, built-in window into the model that convolutional networks do not natively offer.
They are genuinely useful, but they are not a faithful explanation, and this is the central pitfall. Raw attention weights ignore the residual connections that route information around attention, ignore the value vectors that determine what is actually carried, and ignore the MLP — and they compound nonlinearly across layers, so a single layer's map can be misleading. Attention rollout (multiplying attention matrices across layers while accounting for residuals) and gradient-based attributions give more trustworthy maps. The lesson: high attention weight means 'this token was weighted heavily here', not necessarily 'this region caused the output'.
Even so, attention maps have delivered striking findings. In DINO, a ViT trained with self-supervision spontaneously produces class-token attention maps that segment the main object cleanly with no segmentation labels at all — vivid evidence that the model learned object structure. Such results made attention maps a standard tool for sanity-checking, debugging spurious correlations, and communicating what a vision transformer has learned.
Do not equate an attention map with a causal explanation. A model can attend to a region and still not use it, or use a region via residual/value paths while attending elsewhere. For accountability or debugging, corroborate attention maps with perturbation tests or gradient attributions.