Data Leakage
Any path by which information from the test set (or the label) enters training, producing an evaluation that cannot be reproduced online. In neural data it hides in many places: fitting a spatial filter, ICA, CSP, PCA, normalization statistics, artifact thresholds, or a feature-selection ranking on the whole dataset before cross-validation; choosing the time window or frequency band by peeking at test performance; or splitting trials randomly when they are temporally dependent.
The rule is that every data-dependent step must live inside the cross-validation loop, estimated only from each fold's training data and then applied to its test data. Circular analysis — selecting voxels, channels, or time-points by a contrast and then measuring effect size on the same data — is the neuroscience-specific form and inflates effects even under the null hypothesis.
Computing CSP filters on all trials and then cross-validating only the classifier is leakage: CSP has already seen the test labels through the class covariances. Refitting CSP inside each training fold removes it and typically lowers the reported accuracy.
Every label-dependent step must move inside the fold.
Leakage produces results that look excellent offline and collapse online. Because it is invisible in a confusion matrix, the only defense is disciplined pipeline construction and, ideally, a truly held-out session evaluated exactly once.