deep ensembles

The simplest thing that works: train the same network several times from different random initializations and data shuffles, then average their predictions. Because the loss landscape is riddled with many good but different basins, independent runs converge to genuinely different functions that agree where the data is dense and disagree where it is sparse, and that disagreement is a usable uncertainty signal.

Lakshminarayanan and colleagues proposed it as a non-Bayesian alternative, but it is now read as a coarse posterior approximation: each member is a MAP or maximum-likelihood mode, and the ensemble is a uniformly weighted sample over modes, capturing the multi-modal structure that single-mode methods like the Laplace approximation or mean-field variational inference miss. The predictive distribution is the mixture of the members: average the softmax probabilities for classification, the Gaussian outputs for regression.

Empirically deep ensembles are the strongest, best-calibrated uncertainty method on most benchmarks and stay robust under distribution shift, so they are the bar everything else is measured against. The price is linear: M times the training cost, inference cost, and memory. Variants such as snapshot ensembles, batch ensembles, and multi-input/multi-output networks try to recover most of the benefit more cheaply.