overparameterization and interpolation

Give a network far more parameters than there are training examples and it can drive the training loss all the way to zero, fitting every point exactly — interpolating the data. Classical statistics treats this as the danger zone. Modern deep learning finds the opposite: this over-parameterized, interpolating regime is precisely where optimization becomes easy and, with the right implicit bias, where generalization is best.

When parameters vastly outnumber samples, the set of global minima is not a handful of isolated points but a high-dimensional connected manifold, and the loss often satisfies a Polyak-Lojasiewicz-type condition that lets gradient descent reach a global minimum without getting trapped — this is what the neural tangent kernel analysis formalizes. Interpolation by itself would be alarming, but the optimizer does not pick an arbitrary interpolant; its implicit regularization selects a smooth, low-norm one, which is why fitting everything need not mean overfitting.

Overparameterization and interpolation are the connective tissue of modern deep learning theory, tying together double descent, benign overfitting, implicit regularization, and lazy versus feature-learning regimes. The standing caveat is that interpolation is harmless only because of that implicit bias and favorable data geometry, not because memorizing data is intrinsically safe.