JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The practical toolbox: dropout, Laplace, ensembles, last-layer

The cheap, ships-today approximations that dominate practice — MC-dropout, the Laplace approximation, deep ensembles, neural-linear and evidential models — and how to choose among them.

A posterior you already trained

The most painless way into Bayesian deep learning costs almost nothing extra. If your network already uses dropout for regularisation, Monte-Carlo dropout simply leaves dropout on at test time and runs the network several times. Each pass randomly zeroes a different subset of units, so each is effectively a different sub-network — and averaging their predictions, with the spread as your uncertainty, turns out to approximate variational inference over the weights.

p(y^\ast \mid x^\ast,\mathcal{D}) \approx \frac{1}{T}\sum_{t=1}^{T} p\!\left(y^\ast \mid x^\ast, \widehat{W}_t\right),\qquad \widehat{W}_t \sim q_\phi(W)

MC-dropout's predictive is just the average over T stochastic forward passes with dropout left switched on.

A Gaussian around the MAP

The Laplace approximation takes the opposite, post-hoc stance: train your network normally to a MAP solution, then fit a Gaussian posterior around it using the local curvature of the loss — the Hessian, in practice approximated by the Fisher information or a Gauss-Newton matrix. Sharp curvature means a well-determined weight (low variance); flat curvature means an under-constrained one (high variance). You get uncertainty essentially for free from a model you already have.

p(\theta \mid \mathcal{D}) \approx \mathcal{N}\!\left(\theta_{\mathrm{MAP}},\, H^{-1}\right),\qquad H = -\,\nabla_\theta^2 \log p(\theta \mid \mathcal{D})\big|_{\theta_{\mathrm{MAP}}}

The Laplace approximation fits a Gaussian at the MAP, with covariance equal to the inverse Hessian of the negative log-posterior.

The whole game is making that Hessian affordable. The full matrix is quadratic in the number of weights, so people restrict it: diagonal, Kronecker-factored (KFAC) blocks per layer, or — most practical of all — Laplace on the last layer only. Last-layer Laplace has become a go-to recipe precisely because it is cheap, post-hoc, and leaves your trained model's predictions untouched while adding genuine uncertainty.

The embarrassingly strong baseline

Deep ensembles barely look Bayesian: train the same architecture several times from different random seeds, then average. Because the loss landscape is riddled with distinct basins, different seeds land on genuinely different functions; where they agree you trust the prediction, where they disagree the variance flags it. On benchmark after benchmark, a handful of independently-trained networks beats far more elaborate single-model Bayesian methods on both accuracy and calibration.

Why deep ensembles work: different random seeds roll into different basins of the loss landscape, and averaging the networks they reach captures modes no single Gaussian could.

Interactive gradient descent rolling down a loss surface toward the minimum of a basin.

Is it 'really' Bayesian? One honest reading is that the ensemble's seeds are crude samples from the posterior over functions — different modes that a single mean-field Gaussian could never reach. That connection to ensemble learning is why ensembles set the bar epistemic methods must clear. Their cost is the obvious downside: N times the training and N times the memory, which motivates the cheaper tricks below.

Be Bayesian where it counts

A recurring insight: most of a network's expressivity is in its features, but most of its uncertainty can be captured in the head. Neural-linear models act on this — train the deep net as a feature extractor, then place a proper Bayesian linear model (or a Gaussian) over just the last layer. Bayesian linear regression has a closed form, so you get fast, well-grounded uncertainty over learned deep features without ever integrating over the millions of weights below.

p(y^\ast \mid x^\ast,\mathcal{D}) = \int \mathcal{N}\!\left(y^\ast \mid w^\top \phi(x^\ast),\, \sigma^2\right) p(w \mid \mathcal{D})\, dw

Neural-linear freezes the features and does exact Bayesian linear regression only on the last-layer weights.

Evidential deep learning is the most aggressive shortcut of all: train the network to output, in a single forward pass, the parameters of a higher-order distribution (a Dirichlet over class probabilities, or a normal-inverse-gamma over a regression target). The width of that distribution reports uncertainty directly — no sampling, no ensemble. It is gloriously fast, but the elegance hides traps: the uncertainty it reports can be poorly grounded out of distribution and sensitive to the regulariser, so validate it hard before you rely on it.

Choosing in practice

  1. Need a strong default and can afford the compute? A 5-member deep ensemble is the reference; it is hard to beat on accuracy and calibration both.
  2. Already trained one model and want uncertainty post-hoc? Add last-layer Laplace or neural-linear — no retraining, your predictions stay put.
  3. Already use dropout and want a free baseline? Turn on MC-dropout — but trust its uncertainty only relatively, and check it does not flatten far from the data.
  4. Hard latency or memory budget, one forward pass only? Try evidential models — then validate calibration and OOD behaviour adversarially before shipping.