train-test gap

There are two scores you can give a model: how well it does on the examples it studied (training performance) and how well it does on fresh examples it has never seen (test performance). The train-test gap is the difference between them. A small gap means the model genuinely learned the pattern; a large gap means it largely memorized its homework and stumbles on the real exam.

That large gap has a name — overfitting. A student who memorizes the answers to the practice questions aces the practice set but fails the real test, because they learned the specific answers, not the underlying skill. An overfit model does the same: it latches onto quirks and noise in the training data that don't generalize. A near-perfect training score paired with a mediocre test score is the classic fingerprint of overfitting.

Watching the gap is one of the most important habits in machine learning, because only the test score predicts real-world behavior — the training score is, by itself, almost meaningless and easy to inflate. The honest nuance: a tiny gap isn't automatically good news either. If both scores are low, the model is underfitting — too simple to capture the pattern at all. And the test set only stays trustworthy if you don't keep tuning against it; do that, and you slowly overfit to the test set too, and the comforting gap you see becomes a lie. The real goal is a small gap with both scores high — and a final test set you touched exactly once.