Permutation Test

A nonparametric significance test that builds the null distribution of a statistic (for example decoding accuracy) by repeatedly shuffling the labels, re-running the entire cross-validated pipeline, and recording the score under each shuffle. The empirical p-value is the fraction of permutations that meet or exceed the observed score. It makes no Gaussian assumption and, crucially, absorbs whatever optimistic bias the pipeline has — as long as the permutation respects the data's dependence structure.

The subtlety is preserving exchangeability: labels must be permuted at the level of independent units (whole runs or blocks, not individual autocorrelated samples), or the null itself becomes over-optimistic and the test loses validity. Cluster-based permutation extends this to control the family-wise error over many time–frequency–channel points without assuming their correlation structure.