cross-validation
/ KROSS val-ih-DAY-shun /
When you have only a modest amount of data, setting aside a single validation set feels wasteful — and the score you get from one slice can be a fluke. Cross-validation is a clever way to reuse the same data so that every example gets a turn at being the test, giving a steadier estimate of how good a model really is.
The most common recipe is k-fold cross-validation. Shuffle the data and cut it into k equal parts, called folds — say five. Train on four folds and test on the held-out fifth; record the score. Repeat five times, each time holding out a different fold, so every example is tested exactly once and trained on the other times. Average the five scores. That average, and how much the scores wobble around it, tells you both the typical performance and how reliable that estimate is.
Why it matters: a single train/validation split can land lucky or unlucky, especially with small datasets. Cross-validation smooths out that luck and uses the data efficiently. It is most often used to compare models or tune hyperparameters honestly. The cost is computation — you train k times instead of once — and the same anti-leakage care applies: any data cleaning or scaling must be learned inside each training fold, not on the whole set beforehand.
With 500 patients and 5-fold cross-validation, you train five models, each on 400 patients and tested on a different 100. Accuracies come back 81%, 78%, 83%, 79%, 80% — so you report about 80%, give or take 2%, a far more trustworthy number than any single run.
5-fold CV: every example is tested once; averaging tames the luck of a single split.
Cross-validation is for choosing and estimating, not for keeping the final exam honest. Even with CV, you still want a truly untouched test set at the end — and you must fit any preprocessing inside each fold, or you have already leaked.