spectral bias (frequency principle)

Networks learn in a particular order of detail. Early in training they capture the smooth, broad, low-frequency shape of the target function; only later, if at all, do they fill in the fine, rapidly varying high-frequency wiggles. This spectral bias — also called the frequency principle — means a network's first instinct is to fit a simple smooth approximation and refine it gradually, which is itself a form of implicit regularization toward smoothness.

The mechanism is visible through the neural tangent kernel. Decompose training dynamics in the eigenbasis of the kernel; each eigencomponent is learned at a rate set by its eigenvalue, and for standard networks the eigenvalues decay with frequency. Low-frequency components therefore have large eigenvalues and are learned fast, high-frequency ones have tiny eigenvalues and are learned slowly. This explains why plain coordinate networks struggle to represent sharp detail, and why feeding inputs through Fourier features or positional encodings — which boost the high-frequency eigenvalues — lets them fit fine structure.

Spectral bias is one of the most useful lenses on what kind of functions networks prefer, and it directly motivated design choices in neural fields and implicit representations.