neural network Gaussian process (NNGP)

Take a network with random weights and never train it — just look at the outputs it produces across all possible inputs. As the layers become infinitely wide, those outputs become jointly Gaussian: the randomly initialized network is exactly a draw from a Gaussian process. This correspondence, the neural network Gaussian process, turns a deep architecture into a single covariance kernel you can write down in closed form.

The reason is the central limit theorem applied layer by layer. Each preactivation is a sum of many independent random contributions, so in the wide limit it tends to a Gaussian, and the covariance between two inputs propagates through the network by a deterministic recursion fixed by the activation function and the weight variances. Bayesian inference with this kernel reproduces the exact posterior of an infinitely wide Bayesian neural network, with no gradient training involved.

The NNGP is the static, initialization-time and Bayesian counterpart to the NTK. The NNGP kernel describes the prior and posterior over functions; the NTK describes how gradient descent moves a function. Keeping the two straight is the first thing a careful reader of infinite-width theory must do.