model interpretability

Model interpretability is the effort to make a vision model's decisions understandable to humans — to answer 'why did it predict that?' instead of accepting an opaque verdict. Deep networks are black boxes: hundreds of millions of weights transform pixels into a label through computations no human can follow directly. Interpretability methods open the box enough to check whether the model is right for the right reasons, to debug failures, to build justified trust, to satisfy accountability and regulatory demands, and to discover when a model is secretly exploiting a spurious cue rather than the real object.

The most common vision tools are saliency and attribution methods, which produce a heatmap over the input showing which pixels most influenced the decision. Gradient-based methods (vanilla gradients, Integrated Gradients, SmoothGrad) ask how the output changes as each pixel changes; Grad-CAM uses the gradients flowing into the last convolutional layer to highlight the image regions a CNN focused on; perturbation methods (occlusion, LIME, SHAP) instead mask or vary parts of the image and watch how the prediction moves, attributing importance to the parts that matter most. These produce intuitive 'where did it look' maps but are local explanations — they explain one prediction, not the model's global logic.

A second, deeper family asks what concepts a model has learned rather than just which pixels mattered. Concept-based methods (TCAV — Testing with Concept Activation Vectors) quantify how much a human-meaningful concept like 'stripes' or 'wheel' drives a class prediction; feature visualization synthesizes the input that maximally excites a chosen neuron, revealing what it detects; and mechanistic interpretability and probing dissect internal representations to map the network's learned features and circuits. These aim at global, structural understanding of the model, complementing the per-image saliency view, and have become central to understanding large models like CLIP and vision transformers (whose attention maps offer a partial, and sometimes misleading, window).

The hardest lesson of the field is that an explanation can look convincing yet be unfaithful — it may not reflect what the model actually computed. Some saliency methods have been shown to produce nearly the same heatmap even when the model's weights are randomized, meaning they explain the image, not the model; attributions can be unstable (tiny input changes swing the map) and even adversarially manipulable. So interpretability is held to its own evaluation standards: faithfulness (does the explanation track the model's true reasoning?), robustness, and human usefulness — and a plausible-looking heatmap is never, by itself, evidence that the explanation is correct. Interpretability matters most precisely in high-stakes vision (medical diagnosis, autonomous driving, forensic identification) where a wrong-for-the-wrong-reasons model that happens to score well is a liability waiting to surface.