Bayesian model averaging (BMA)

When several models could explain your data, Bayesian model averaging refuses to pick one. Instead of selecting the single best hypothesis, it makes predictions by averaging over all of them, weighting each by how plausible it is given the data, namely its posterior probability. A model that fits well and is a priori reasonable gets a large vote; an overfit or implausible one gets a small one.

The predictive distribution is the integral of each model's prediction times the posterior over parameters or models. This is the genuinely Bayesian answer, and the thing all the methods in this field are trying to approximate: MC dropout, deep ensembles, SGLD samples, and Laplace samples are each a way of drawing the parameter settings you then average over. Crucially, BMA is not model combination to build a better single model; it is marginalization that reflects honest uncertainty about which model is right.

Wilson and Izmailov argue that the reason Bayesian deep learning helps is precisely this marginalization over a multi-basin posterior, and that deep ensembles are a surprisingly good BMA. A subtlety: under misspecification the true Bayes posterior can concentrate on a single model and average poorly, which connects to the cold posterior effect, where a tempered posterior sometimes averages better than the exact one.