saliency map
/ SAY-lee-un-see map /
A saliency map is a heatmap laid over an input — usually an image — showing which parts the model paid most attention to when it made its decision. Where the model looked hardest glows bright; where it ignored stays dark. For a model that labeled a photo "dog," a good saliency map would, ideally, light up the dog and leave the lawn dim. It is the most visual, intuitive form of explanation, and that vividness is both its strength and its trap.
The simplest version asks: if I nudge each pixel slightly, how much does the model's answer change? Pixels that swing the answer most are the most "salient." This is computed from the model's gradients — the same derivatives used in training. Many fancier variants exist (smoothing the noise, integrating along a path, using internal activations), each trying to produce maps that are cleaner and more faithful to what the model actually used.
Saliency maps matter because a bright blob over the right object is reassuring, and a bright blob over the background is an alarm — like the wolf-versus-snow case. But the field has learned hard humility here. Some popular saliency methods produce nearly identical-looking maps even when the model's weights are randomized into nonsense, meaning the pretty picture wasn't really explaining the trained model at all. A saliency map can be a genuine window or a Rorschach blot; you cannot tell which just by looking at how plausible it seems.
A chest X-ray model flags 'pneumonia.' Its saliency map glows brightly — but on the corner of the image, where a hospital's printed metadata tag sits, not on the lungs. The model had learned that one hospital (which saw sicker patients) used a particular tag, and was reading the label, not the disease. The heatmap caught a shortcut a metrics table never would.
A saliency map can reveal a model 'reading the label, not the lungs.'
Run the sanity check before you trust a saliency map: if randomizing the model's weights barely changes the map, the map is reflecting the input or the method's own quirks, not the trained model — and explains nothing. A pretty, intuitive heatmap is not automatically a faithful one.