early stopping

If you train a model for too long it eventually stops learning the general pattern and starts memorizing the noise in the training set — overfitting — and its performance on new data quietly gets worse even as its training loss keeps improving. Early stopping is the disarmingly simple remedy: watch how the model does on a held-out validation set during training, and stop as soon as that validation performance stops improving, rather than running for a fixed, possibly excessive number of epochs. You halt the model at the sweet spot, after it has learned the signal but before it has begun fitting the noise.

In practice you evaluate a validation metric (loss or accuracy) at regular intervals and track the best value seen so far. Because the metric is noisy and can dip and recover, you do not stop at the very first uptick; instead you use a patience parameter — the number of evaluations you are willing to wait without improvement before giving up — and when patience is exhausted you stop and restore the weights from the best checkpoint, not the final, slightly-overfit ones. Patience and the evaluation interval are the two knobs: too little patience stops prematurely on noise, too much wastes compute and lets some overfitting creep in.

There is a satisfying theoretical reason early stopping works: it is a form of implicit regularization. Each gradient step lets the weights move a little further from their small, random initial values, so stopping early keeps the effective magnitude of the weights bounded — much as weight decay does explicitly. For simple linear models this connection is exact, and more generally, limiting the number of optimization steps limits how much the model can specialize to the training set, which is precisely capacity control by another name.

Early stopping is cheap, requires no extra hyperparameter strength to tune beyond patience, and composes with every other regularizer, so it is a near-universal default. The main caveats are that it needs a trustworthy validation set held out from training (and not also used for tuning everything else, or you overfit to it), and that it interacts with learning-rate schedules: a cosine schedule, for instance, is designed to run for a planned number of steps and reach a low final learning rate, so stopping it 'early' means it never reaches its intended end state — in such setups early stopping is often used as a safety net rather than the primary stopping criterion.