The selection problem
Guide 1 left us with a selection problem: among infinitely many zero-loss interpolants, training reliably picks ones that generalize. The hypothesis at the heart of this section is implicit regularization by gradient descent — the optimizer itself, with no explicit penalty term, biases solutions toward a preferred subset. The explicit regularizers you know (weight decay, dropout, early stopping) help, but they are not the main event; even without any of them, plain gradient descent generalizes.
The cleanest theorems live in simple models. For separable logistic regression, gradient descent's direction converges to the max-margin solution — the same one an SVM would pick — even though the loss has no margin term. For matrix factorization, gradient flow from small initialization is biased toward low nuclear norm. These are exact statements: 'GD secretly minimizes norm X subject to fitting the data.'
Two classes of points separated by the widest possible margin, with support vectors lying on the margin.
Flat minima generalize better
A complementary lens looks at the shape of the basin. The flat minima hypothesis says solutions sitting in wide, flat regions of the loss landscape generalize better than those in sharp, narrow ones. Intuition: a flat minimum is robust — small perturbations of the weights (which is what train/test distribution shift effectively causes) barely change the loss, so the test loss stays close to the training loss. Sharpness is often measured by the top eigenvalues of the loss Hessian.
This is more than a metaphor. Sharpness-Aware Minimization (SAM) turns it into an objective: minimize the worst-case loss in a small neighborhood, explicitly seeking flatness, and it measurably improves generalization. The honest caveat: naive sharpness is not reparameterization-invariant (you can rescale weights to fake flatness), so the precise notion that controls generalization is still debated — adaptive and scale-invariant sharpness measures are an active repair.
Sharpness-Aware Minimization replaces the loss with its worst case in a ρ-ball, explicitly seeking flat minima.
The edge of stability
Classical optimization says full-batch gradient descent is stable only when the learning rate η stays below 2/λ, where λ is the top Hessian eigenvalue (the sharpness). Above that, a quadratic bowl makes you diverge. Watch a real network train and something stranger happens: sharpness rises until λ reaches almost exactly 2/η, then hovers there — the loss bounces non-monotonically yet keeps trending down over the long run. This is the edge of stability.
Classical descent is stable only below 2/λ; networks instead hover just past this threshold at the edge of stability.
The lesson is that gradient descent at practical learning rates is not a discretized gradient flow — it lives in a genuinely discrete, mildly unstable regime that the small-step continuous theory cannot see. And this instability is productive: the very mechanism that caps sharpness at 2/η is pushing the model toward flatter regions. Optimization and the flat-minima story are the same story told from two ends.
Spectral bias: simple before complex
There is also an implicit bias in time. The spectral bias, or frequency principle says networks fit the low-frequency, smooth components of the target function first, and only later — if at all — the high-frequency wiggles. Through the NTK lens this is exact: components aligned with large kernel eigenvalues are learned at a rate set by those eigenvalues, and for standard networks the smooth directions dominate the spectrum.
Spectral bias is a double-edged inductive bias. It is why early stopping works (you halt before the noisy high frequencies are fitted) and why networks are naturally smooth interpolators that generalize. It is also why plain MLPs struggle to represent sharp, high-frequency targets — the fix in fields like neural rendering is to inject high frequencies explicitly (Fourier features, positional encodings) so the spectral bias has something to grab.