implicit regularization of gradient descent

When a model has more parameters than data, infinitely many settings drive the training loss to zero, yet gradient descent does not pick one at random. Left with no explicit penalty, the optimizer still favors particular solutions — typically small-norm, large-margin, or flat ones. This silent preference, baked into the dynamics rather than the objective, is the implicit regularization of gradient descent, and it is a leading candidate for why unregularized networks generalize.

The bias can sometimes be pinned down exactly. Gradient descent on separable logistic regression converges in direction to the maximum-margin (minimum L2-norm) classifier. On matrix factorization it drifts toward low nuclear norm, an implicit low-rank prior. The specific norm that gets minimized depends delicately on the loss, the parameterization, the initialization scale, and even the step size — there is no single universal quantity that gradient descent always shrinks.

This reframes generalization in deep learning: the architecture and the algorithm, not an added regularizer, supply the inductive bias. The honest qualifier is that outside a handful of solvable models the implicit bias is only partially characterized, so it remains more a guiding principle than a closed theory.