cold posterior effect

A puzzle at the heart of Bayesian deep learning. In theory, predicting with the true Bayesian posterior should be optimal. In practice, Wenzel and colleagues found that for deep networks you get better accuracy and likelihood by sharpening the posterior, raising it to a power one over T with temperature T below one, effectively trusting the data more than Bayes says you should. The cold posterior, with T below one, beats the genuine one at T equal to one, which is unsettling.

The tempered posterior is the Bayes posterior raised to the power one over T: at T equal to one it is exact Bayes, and as T tends to zero it collapses to the MAP estimate. The effect is the observation that test performance often improves monotonically as T drops below one. Proposed explanations include data augmentation making the effective likelihood count each point many times so the real T is implicitly below one, the priors we put on weights being bad and over-regularizing, the likelihood being misspecified, and dataset curation breaking the i.i.d. generative assumption. Which cause dominates is still debated.

The cold posterior effect matters because it questions whether our approximate-inference wins come from being Bayesian or from accidentally down-weighting a bad prior. Later work argues that much of it is an artifact of data augmentation and curation rather than a deep failure of Bayes, but it remains the standard cautionary tale about taking the deep-net posterior at face value.