function-space inference

A neural network has millions of weights, but you never actually care about the weights; you care about the function they compute. Function-space inference takes this seriously: instead of placing a prior on weights and reasoning in that huge, redundant, hard-to-interpret space, it places the prior and does inference directly over the functions the network can represent, so that two different weight settings computing the same function are correctly treated as the same object.

The motivation is that a weight-space prior such as a Gaussian on the parameters induces an implicit, opaque, often pathological prior over functions, and KL divergences in weight space are misaligned with what we actually want to control. Function-space methods instead constrain or match the distribution of network outputs at chosen measurement points, using a Gaussian-process view, functional KL or variational objectives (fVI), or function-space regularization toward a GP prior. This also fixes the over-parameterization symmetry that makes weight posteriors non-identifiable.

It connects deep nets to Gaussian processes, exactly so in the infinite-width or neural-tangent-kernel limit, and gives more meaningful priors and better-behaved continual learning. The hard part is that the functional KL between stochastic processes is generally infinite or intractable, so practical methods rely on finite sets of measurement points and approximations whose correctness is subtle.