mean-field theory of neural networks

Instead of tracking each neuron in a wide two-layer network individually, picture them as a large population and ask how the whole population shifts during training. In the mean-field limit the relevant object is the distribution of neuron parameters, and learning becomes the flow of that distribution — like a cloud of particles drifting under the force exerted by the loss. The single network is replaced by a probability measure over neurons.

The mechanism turns on a scaling choice: weight each neuron's output by one over the number of neurons, not one over its square root. Under this normalization, as the width grows the empirical distribution of parameters evolves according to a Wasserstein gradient flow — equivalently a nonlinear continuity-equation PDE — that decreases the loss viewed as a functional of the distribution. Crucially, the neurons genuinely move, so the features adapt during training.

That is the decisive contrast with the neural tangent kernel. The NTK's lazy regime freezes the features; the mean-field regime lets them learn, which is why it can prove global convergence while modeling the feature-learning that makes deep networks effective. The trade-off is that mean-field analysis is sharp mainly for two-layer architectures.