Interpretability of neural decoders

Interpretability methods try to explain what a neural decoder has learned, which matters in BCI both for scientific insight (does the model use physiology or artifact?) and for safety and regulatory trust. Techniques include gradient- and perturbation-based saliency (which time points, channels or frequencies the output depends on), activation-maximization and learned-filter visualization (Schirrmeister's frequency-domain analysis of ConvNet filters), ablation studies, and — for models linear in their features — the crucial Haufe transformation that converts a discriminative filter weight into an interpretable activation pattern.

The Haufe insight is a recurring trap: the weights of a linear (or linearized) decoder do not indicate where the signal is, because large weights are often assigned to channels carrying noise the classifier must suppress. Only the corresponding activation pattern (the weight vector multiplied by the feature covariance) is neurophysiologically interpretable. Beyond this, saliency maps for deep networks are notoriously fragile and can pass sanity checks while being unfaithful, so interpretability claims in BCI warrant the same skepticism as decoding claims.