overfitting

A model overfits when it learns the training data too well — including its random noise and quirks — and as a result performs worse on new, unseen data. The classic image is a student who memorizes the exact answers to last year's exam rather than understanding the subject: flawless on the questions they have seen, lost on anything new. An overfit model has effectively memorized its training set instead of learning the general pattern, so its impressive training accuracy is a mirage that does not transfer to the real test that matters.

The telltale signature is a diverging gap between training and validation performance over time. Early in training both the training loss and the validation loss fall together. At the onset of overfitting they part ways: the training loss keeps dropping toward zero while the validation loss bottoms out and then starts climbing back up. That rising validation curve, even as training loss improves, is the canonical diagnostic — the model is now spending its capacity fitting noise that exists only in the training set.

Overfitting is the opposite failure from underfitting, and distinguishing them tells you what to do. Underfitting (high bias) is when the model is too simple or undertrained to capture even the training pattern, so both training and validation performance are poor and close together; the remedy is more capacity, more training, or better features. Overfitting (high variance) is when the model is too flexible relative to the data, so training performance is excellent but validation lags; the remedy is to constrain the model or feed it more information. This is the bias-variance tradeoff: too little flexibility underfits, too much overfits, and the art is finding the middle.

The remedies are essentially the rest of this field's toolbox: get more training data (the most powerful fix); apply data augmentation to simulate more data; add regularization (weight decay, dropout); reduce model size or use early stopping to limit how much the model can specialize to the training set; and use transfer learning when data is scarce. One important modern caveat: the textbook 'more parameters always overfit more' picture is incomplete. Hugely overparameterized deep networks often generalize well even after fitting the training data perfectly — the 'double descent' phenomenon — so capacity alone does not determine overfitting; the data, regularization, and optimization all matter.