The one-number problem
A trained network gives you one weight vector and, for each input, one prediction. Ask it about a digit it has never seen — a photo of static, a corrupted scan, a class that did not exist at training time — and it answers with the same breezy confidence it shows on the training set. The softmax outputs look like probabilities, but they are the probabilities of a single, arbitrarily-chosen function. They tell you nothing about whether that function was the right one to pick.
This is the gap maximum likelihood training leaves open. Minimising a loss finds a good fit; it never quantifies how many other good fits exist, nor how wildly they would disagree away from the data. Whenever a model must abstain, defer to a human, gather more labels, or act in the world, that missing quantity — *how much should I trust this?* — is the whole game.
A distribution over weights
The Bayesian move is to stop treating the weights as a number to be optimised and start treating them as a random variable to be inferred. Put a prior p(w) over weights, write the likelihood p(D|w) as your usual loss in disguise, and the posterior p(w|D) follows from Bayes' theorem. A Bayesian neural network is exactly this: not one network, but a posterior distribution over networks, each weighted by how well it explains the data and how plausible it was a priori.
Interactive Bayesian belief update sliding a prior distribution into a posterior as evidence arrives.
Prediction is then an average, not a single pass. To predict y for a new x you integrate the predictions of every plausible network against the posterior — the posterior predictive distribution. Where the data pin the weights down, all those networks agree and you get a sharp, confident answer. Where the data are silent, they fan out, and that disagreement is your uncertainty. Notice that ordinary training with weight decay already computes the peak of this posterior: MAP estimation is Bayes with the integral replaced by its single highest point.
Prediction becomes an average over every plausible network — the posterior predictive integrates each network's output against the posterior.
Two kinds of not-knowing
Not all uncertainty has the same cure, and conflating the two is the most common conceptual mistake in the field. Epistemic uncertainty is uncertainty about the model — which function is right — and it is reducible: collect more data in the right region and it shrinks toward zero. Aleatoric uncertainty is the irreducible noise in the data itself — a blurry image, an inherently ambiguous label, sensor jitter — and no amount of extra data makes it go away.
The distribution over weights captures the epistemic part: away from the data the networks disagree, so epistemic uncertainty is large — precisely the signal you want for out-of-distribution detection and active learning. The aleatoric part lives in the likelihood instead: you let the network output its own noise level (a predicted variance, or the concentration of a distribution) so it can say 'this input is just intrinsically noisy.' Good uncertainty quantification reports both, separately.
Why exact Bayes is hopeless here
Bayes' theorem has a denominator: the marginal likelihood, an integral of the likelihood against the prior over all weight settings. For a conjugate toy model this integral is a clean formula. For a network with millions of weights it is a high-dimensional integral over a wildly multimodal, non-conjugate landscape — utterly intractable. Every method in this track is, at heart, a different way to dodge that integral.
Bayes' theorem for the weights — the denominator is the marginal likelihood, an integral over all weight settings that is intractable for a real network.
There are two grand strategies. Approximate the posterior with something tractable you can write down and optimise — a Gaussian, a flow, a low-rank curvature — which is fast but only as honest as the approximating family. Or sample from the posterior with a Monte-Carlo chain that needs no closed form but pays in compute and mixing time. Most of what works in practice is a clever, cheap compromise between the two.
The road ahead
This track climbs that compromise. Guide 2 makes the approximation route precise with variational inference — the workhorse. Guide 3 collects the cheap-and-cheerful approximations people actually ship: MC-dropout, the Laplace approximation, and the embarrassingly strong deep ensembles. Guide 4 takes the sampling route and then the radical step of inferring over functions instead of weights. Guide 5 asks whether any of it can be trusted, with calibration, conformal guarantees, and the open puzzles.