grokking (delayed generalization)

Grokking is a startling training curve. The model memorizes the training set almost immediately — training accuracy hits one hundred percent while test accuracy sits at chance — and then, after a long plateau during which nothing seems to happen, sometimes a hundred times more steps later, test accuracy abruptly snaps to near-perfect. Generalization arrives long after fitting, as if the network suddenly understood the rule.

It was first reported on small algorithmic tasks such as modular arithmetic, where there is a clean underlying rule to discover. A leading account is that weight decay slowly pushes the network off a high-norm memorizing solution toward a lower-norm, structured generalizing one — for modular addition the learned solution turns out to implement a Fourier or circular representation. Mechanistically it looks like a competition between a memorizing circuit and a generalizing circuit, with the latter winning only after sustained pressure.

Grokking is a vivid, controlled window into the gap between memorization and generalization. The honest caveat is that it is sensitive to regularization strength and the fraction of data used, and the precise mechanism remains an active research question rather than a closed account.