Why Overparameterized Networks Generalize

The textbook picture, and how it breaks

Classical statistical learning tells a tidy story. As you grow a model's capacity, training error falls monotonically while a generalization gap opens up; the bias–variance tradeoff says test error is U-shaped, minimized at some sweet spot of capacity before overfitting takes over. The advice that follows is to never let a model interpolate noisy data — fit it exactly and you have memorized the noise.

Modern deep learning ignores this advice and thrives. State-of-the-art networks routinely have far more parameters than training examples, drive training loss to zero, fit randomly relabeled data perfectly — and still generalize beautifully on real labels. The classical theory is not wrong; it is incomplete. It describes a regime we have largely left behind.

Overparameterization and interpolation

Call a model overparameterized when it has enough free parameters to interpolate the training set — to drive training loss to (near) zero. The phenomenon of overparameterization and interpolation is the starting point of deep learning theory: there is a whole manifold of weight configurations that all achieve zero training loss, and they differ wildly in how well they generalize. Optimization does not just find a solution; it selects one from a huge set, and that selection is everything.

This reframes the central question. It is no longer 'does the model have too much capacity?' but 'among all the zero-training-loss solutions, why does gradient-based training land on ones that generalize?' The classical empirical risk minimization story has no answer, because every interpolating solution has the same (zero) empirical risk. The answer must come from an implicit preference encoded by the architecture and optimizer — a theme we develop across this track.

\theta^{\star} = \operatorname*{arg\,min}_{\theta}\ \lVert \theta \rVert \quad \text{s.t.} \quad f_{\theta}(x_i)=y_i,\ i=1,\dots,n, \qquad p \gg n

Among all parameter settings that interpolate the data, gradient training implicitly selects the minimum-norm one.

Double descent: the U-curve was only half the story

Extend the capacity axis far past the interpolation threshold and a striking shape appears. Test error follows the classical U up to the point where the model first fits the data exactly — the interpolation threshold, where error often spikes — and then, instead of exploding, it falls again, frequently to a new global minimum below the classical sweet spot. This is double descent, and it appears across model size, training time, and dataset size.

\gamma=\frac{p}{n}: \quad \mathrm{Risk}(\gamma)\ \text{peaks at}\ \gamma=1\ (p=n),\ \text{then descends again as}\ \gamma\to\infty

Test risk peaks exactly at the interpolation threshold p = n, then falls a second time — the double descent.

Intuition: right at the threshold, there is exactly one way to fit the data, and the model is forced into a contorted, high-norm interpolant that is wildly sensitive to noise. Push past the threshold and you regain a choice of interpolants; the optimizer can now select a smooth, low-norm one. More parameters buy not more memorization but more room to find a simple fit. The same logic explains 'epoch-wise' double descent over SGD training time.

Benign overfitting: interpolating noise without paying for it

Push the idea to its sharpest form. Take labels with genuine noise, fit them to exactly zero training error, and ask what happens to test error. Classically this is fatal — you have memorized the noise. Yet in high dimensions, overparameterized models can absorb the noise into directions that barely affect predictions on new points. This is benign overfitting: the interpolant spikes locally around each noisy training point but stays smooth almost everywhere else, so the noise-fitting is harmless.

The theory (worked out cleanly for linear regression and kernels) pins benign overfitting to the spectrum of the data: it occurs when there are many low-variance directions that can soak up noise without distorting the signal subspace. Whether it is benign, tempered, or catastrophic depends on how fast those eigenvalues decay — a precise statement that connects interpolation to the geometry of the feature space.

\mathbb{E}\,[\,R(\hat{\theta})-R^{\star}\,]\ \lesssim\ \underbrace{\lVert\theta^{\star}\rVert^{2}\,\tfrac{1}{n}}_{\text{bias}}\ +\ \underbrace{\sigma^{2}\!\left(\frac{r_{0}(\Sigma)}{n}+\frac{n}{R_{n}(\Sigma)}\right)}_{\text{variance from noise}}

Excess risk splits into a bias term plus a noise term that stays small when the covariance spectrum has a long low-variance tail.

Reframing the field's question

Pulling the threads together gives the agenda for the rest of this track. The classical theory was never falsified; it was outgrown. The real questions are sharper:

What does training implicitly prefer among the infinitely many interpolating solutions? (Guide 3, implicit regularization.)
Can we describe trained networks exactly in any tractable limit? (Guide 2, infinite-width kernels.)
What is the geometry of the solution set, and is it surprisingly simple? (Guide 4, connectivity and collapse.)
What strange dynamics happen along the way — and do classical bounds even apply? (Guide 5, frontier phenomena.)