Why a single split lies
With 150 trials, one train/test split gives a test accuracy whose confidence interval is enormous — a swing of ±10 points from luck alone is routine. Cross-validation averages over K splits so that every trial is tested exactly once, turning a single noisy number into an estimate of risk: the decoder's expected error on unseen data drawn from the same distribution.
\hat R_{\mathrm{CV}}=\frac{1}{K}\sum_{k=1}^{K}\frac{1}{|\mathcal{D}_k|}\sum_{i\in\mathcal{D}_k} L\!\big(y_i,\ \hat f_{-k}(x_i)\big)The CV risk estimate. Crucially \hat f_{-k} is trained without fold k — the estimate is only valid if the fold's test trials influenced nothing in the model.
To estimate real-world error honestly, hold out a chunk of data, train on the rest, test on the held-out chunk, then rotate through all chunks — so every trial is judged by a model that never saw it.
- K
- Number of folds the data is split into.
- \mathcal{D}_k
- The trials in fold k — the test set for that round.
- \hat f_{-k}
- The model trained on everything except fold k.
- L
- The loss — e.g. whether the prediction was wrong.
With K=5 you train five times, each on 80% of the data and testing on the untouched 20%. Any peek at the test fold is data leakage. See cross-validation.
The formula hides the entire difficulty in one subscript: \hat f_{-k} must be built with zero knowledge of fold k. Every leakage bug is a violation of that subscript.
The leakage zoo
Data leakage is any path by which test information reaches the trained model. In neural data it is endemic because so many pipeline steps are fit from data. Here are the ones that actually bite:
- Preprocessing on the full set: fitting ICA, a spatial filter, CSP, or a z-score normaliser on all trials before splitting — the filter has already seen the test trials.
- Selecting features or hyperparameters on test performance, then reporting that same performance (Guide's nested fix).
- Temporal overlap: sliding windows or epochs that share samples straddle the train/test boundary and copy information across it.
- Block leakage: trials from one continuous block landing in both train and test, so the model learns that block's drift rather than the task.
- Subject leakage in 'cross-subject' claims: the same participant's data in both sets inflates apparent transfer.
Nested CV: honest hyperparameter selection
The moment you tune anything — a shrinkage grid, a filter bank, a network's learning rate — the tuned score is optimistically biased, because you chose the setting that looked best on that very data. Nested cross-validation restores honesty with two loops: an inner loop selects hyperparameters, an outer loop measures the whole selection procedure on data the inner loop never saw.
- Split data into K outer folds.
- Hold out outer fold k; on the remaining data run an inner CV over the hyperparameter grid.
- Refit with the inner-best hyperparameters on all of the outer-train data.
- Score once on the untouched outer fold k; average these outer scores as the honest performance estimate.
Time is not exchangeable
Standard CV assumes trials are exchangeable draws from one distribution. Neural signals violate this: impedance drifts, electrodes settle, attention wanes, so the distribution moves through the session. This is covariate shift, a form of neural non-stationarity. Random K-fold, by mixing early and late trials, quietly lets the decoder interpolate across drift and overestimates deployment accuracy.
The realistic protocol is chronological: train on earlier trials, test on strictly later ones, mirroring how the decoder will actually be calibrated once and used going forward. The gap between random-fold and chronological accuracy is itself a useful diagnostic — a large gap means your decoder leans on non-stationary structure that will not survive to tomorrow's session.