JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Infinite Width: Gaussian Processes and the Neural Tangent Kernel

Send the width of a network to infinity and the math collapses into something we can solve exactly — a Gaussian process at initialization, kernel regression during training. We build the NNGP and NTK and ask what they leave out.

Why take width to infinity

Finite networks are nonlinear, non-convex, and tangled. But many quantities concentrate as you add neurons: a sum of many independent random contributions, suitably scaled, behaves like its average plus a Gaussian fluctuation. The infinite-width limit is the physicist's move of taking a large-N limit to make a messy system exactly solvable. The reward is two clean objects — the NNGP kernel and the neural tangent kernel — that turn a neural network into linear algebra.

Two scalings give two limits. The NTK / lazy scaling keeps per-layer outputs O(1) and yields kernel behavior; the mean-field scaling keeps the function O(1) while letting individual neurons move, and yields genuine feature learning. Which limit you take is a modeling choice with very different predictions — we will meet both.

The wide multilayer network whose hidden width we send to infinity — each scaling treats these per-layer sums differently.

A multilayer perceptron with an input layer, two hidden layers, and an output, connected by weighted edges.

NNGP: a network at initialization is a Gaussian process

Take a deep network with random initialized weights and look at its output as a function of the input. As width goes to infinity, the central limit theorem kicks in layer by layer: the output is a draw from a Gaussian process whose covariance — the NNGP kernel — is computed by a simple recursion through the layers. This is the neural-network Gaussian process correspondence. Bayesian inference with an infinitely wide network is exactly GP regression with the NNGP kernel.

K^{(\ell)}(x,x') \;=\; \sigma_w^2\,\mathbb{E}_{f\sim\mathcal{N}\!\left(0,\,K^{(\ell-1)}\right)}\!\left[\phi\!\left(f(x)\right)\phi\!\left(f(x')\right)\right] \;+\; \sigma_b^2

The NNGP kernel as a layerwise recursion: each layer's covariance is an expectation of activations under the previous layer's Gaussian.

K0 = x @ x.T              # input covariance
for layer in range(depth):
    K = activation_covariance(K0)   # e.g. arccos kernel for ReLU
    K0 = K
f_post = gp_posterior(K, X_train, y_train, X_test)   # NNGP regression
The NNGP kernel as a layerwise recursion; predictions are closed-form GP posteriors.

NTK: training becomes kernel regression

Now train. The neural tangent kernel is the inner product of the network's gradients with respect to its parameters — built from the Jacobian of outputs over weights. The key infinite-width fact: as width grows, this kernel stops moving during training. The weights barely budge from initialization (the 'lazy' regime), the network behaves like its first-order Taylor expansion, and gradient descent on the squared loss becomes exactly kernel gradient flow with the fixed NTK.

\Theta(x,x') \;=\; \nabla_{\theta} f(x;\theta)^{\top}\,\nabla_{\theta} f(x';\theta)

The neural tangent kernel: the inner product of the network's parameter-gradients at two inputs.

This is a remarkable gift. A linear ODE replaces a non-convex optimization; convergence to zero training loss is guaranteed and you can write the test prediction in closed form. The eigenvalues of the NTK even predict which components of the target are learned fast versus slow — directly foreshadowing the spectral bias we meet in Guide 3.

The mean-field alternative: where features actually move

The NTK regime is also its own indictment: if weights never move, the network never learns new features — it is a fixed kernel machine, and its expressive power is whatever the initial kernel already had. Real networks plainly do learn features. The mean-field theory of neural networks takes the other limit: a two-layer net is viewed as a distribution over neurons, and training is a gradient flow (a Wasserstein flow) of that distribution. Neurons migrate, specialize, and the kernel evolves — feature learning is built in.

f(x) \;=\; \frac{1}{N}\sum_{i=1}^{N} a_i\,\phi\!\left(w_i^{\top}x\right) \;\xrightarrow[\;N\to\infty\;]{}\; \int a\,\phi\!\left(w^{\top}x\right)\,\rho(da,\,dw)

The mean-field view: the output becomes an integral over a distribution of neurons that actually moves during training.

What the kernel picture explains, and what it misses

  1. Explains: guaranteed convergence of overparameterized training, closed-form generalization, and why initialization scale matters.
  2. Misses: real networks beat their own NTK on hard tasks — feature learning gives a measurable edge the fixed kernel cannot reach.
  3. Misses: transfer learning, representation reuse, and the discrete jumps of Guide 5 all live outside the lazy regime.

So treat infinite width as a solvable model organism: the e. coli of deep learning theory. It is not your trained ResNet, but it is the first setting where you can prove things end to end, and every richer theory is measured against the gap between it and reality. Keep that gap — feature learning — firmly in view as we turn to what finite, non-lazy gradient descent actually prefers.