benign overfitting

Train a model until it fits every training point exactly, label noise and all — the textbook recipe for disaster — and then watch it predict new data almost perfectly anyway. This is benign overfitting: memorizing the noise turns out to cost nothing on the test set. It directly contradicts the classical intuition that zero training error on noisy data must mean ruined generalization.

The explanation lives in high-dimensional geometry. The minimum-norm interpolator must pass through the noisy labels, but it can absorb that noise into a vast number of low-variance, nearly irrelevant directions of the data covariance, so each direction carries only a sliver of the damage. When the covariance spectrum has the right shape — many small directions to soak up noise, a few large ones that carry signal — the interpolation is harmless. Bartlett and collaborators made this precise for linear regression with sharp conditions on the effective ranks of the covariance.

Benign overfitting is one of the cleanest theoretical bridges to why over-parameterized networks generalize despite fitting everything. The caveat is that benignity is conditional: change the spectrum and the very same interpolation becomes harmful, so it is a property of the data geometry, not a universal license to interpolate noise.