double descent

Classical statistics draws a U-shaped curve: as you add capacity, test error first falls then rises, and you are warned never to overshoot. Double descent says the story does not end there. If you keep growing the model past the point where it can fit the training data exactly — the interpolation threshold — the test error rises to a sharp peak and then, surprisingly, comes down again, often to its lowest value of all.

The peak sits where the number of effective parameters roughly equals the number of training samples. There the interpolating solution is forced and brittle, so variance explodes. Beyond the threshold there are many ways to fit the data perfectly, and the optimizer's implicit bias selects a smooth, low-norm one, which generalizes well. The same shape appears not only across model size but also along training time (epoch-wise) and dataset size (sample-wise).

Double descent reconciles the classical bias-variance picture with the modern observation that enormous networks generalize. It is not a contradiction of the U-curve but an extension of it: the U-curve is the under-parameterized half, and a second descent governs the over-parameterized half.