Cross-Validation on Neural Data
Cross-validation estimates out-of-sample performance by repeatedly holding out part of the data, training on the rest, and averaging test scores across folds. It is the workhorse of BCI evaluation, but its central assumption — that samples are independent and identically distributed — is systematically violated by neural data: temporally adjacent trials share slow drifts, artifacts, and autocorrelation, so a random k-fold split leaks information between train and test and inflates accuracy.
The remedy is to split along the true unit of independence: leave-one-run-out or leave-one-block-out (respecting recording blocks), and, when the deployment target is a future session, cross-session or leave-one-session-out validation. Any reported cross-validated accuracy should name the fold structure, because a within-run random split and a cross-session split can differ by tens of percentage points.
A motor-imagery decoder scoring 92% under random 10-fold CV dropped to 71% under leave-one-run-out on the same data; the gap was slow drift shared within runs that the random folds let the classifier memorize.
The fold structure decides whether the estimate is honest.
Blocked or temporal splitting is not conservative pedantry — it is the only estimate that predicts real online use. Random k-fold on continuous neural data is one of the most common sources of published over-optimism.