flat minima and generalization

Among the many weight settings that reach the same low training loss, not all are equal at test time. A persistent empirical regularity is that solutions sitting in a wide, flat valley of the loss generalize better than those wedged in a sharp, narrow crevasse. Intuitively, a flat minimum is robust: small perturbations to the weights — or the inevitable mismatch between the training loss and the true loss — barely move the loss, so the solution is forgiving.

Flatness is typically measured by how small the Hessian eigenvalues are, or how much the loss rises under a worst-case weight perturbation. Several theories link it to generalization: minimum-description-length and Bayesian arguments give flat minima a larger Occam factor (they occupy more volume in weight space), and PAC-Bayes bounds make perturbation-robustness an explicit term controlling the generalization gap. This view motivated training methods that actively seek flatness, notably sharpness-aware minimization and entropy-SGD.

The sharp caveat is that flatness is not reparameterization-invariant. Rescaling the weights of a ReLU network can turn a sharp minimum into a flat-looking one without changing the function at all, so naive sharpness can be gamed. Meaningful theory therefore needs scale-aware or normalized measures of flatness.