variational inference
/ vair-ee-AY-shun-ul IN-fur-uns /
Variational inference is a way to get an approximate answer to a Bayesian problem when the exact one is out of reach. The true posterior — the full landscape of belief after seeing data — is often a tangled shape you cannot compute. Instead of giving up, variational inference picks a simpler, well-behaved shape from a family you can handle (say, a smooth bell curve) and dials its settings until it hugs the true posterior as closely as possible. It turns a hard integration problem into an optimization problem: not 'sum over everything' but 'tune knobs to fit.'
The trick for measuring 'as closely as possible' is a quantity that scores how far the simple stand-in sits from the truth, and the algorithm slides the knobs downhill to shrink that gap. Because optimization is something computers do fast and at scale, variational inference can fit enormous models on huge datasets where sampling methods would crawl. This speed is its main selling point and the reason it sits at the heart of modern tools like the variational autoencoder.
Why it matters, told honestly: variational inference trades exactness for speed, and you should know what you're trading. Because it forces the answer into a simpler family, it can miss the true shape — most notably, the popular versions tend to be overconfident, reporting a posterior that is too narrow and understating how uncertain you really are. It also only finds a local best fit, not a guaranteed global one. It is a superb practical tool when you need a fast, scalable approximation, as long as you don't read its tidy, confident output as the exact truth.
Imagine the true posterior is a banana-shaped smear over two parameters. Exact computation is hopeless. Variational inference approximates it with a simple round blob (a Gaussian), sliding and resizing the blob to overlap the banana as much as it can. The blob can sit nicely on the banana's fat middle but, being round, it can't follow the curve into the tips — so it understates the spread.
A simple shape can't perfectly trace a complex posterior, which is why variational inference often underestimates uncertainty.
The most important caveat: the common form of variational inference tends to be overconfident, producing posteriors that are too tight. If you use it and then quote its uncertainty as if exact, you will systematically claim to be more sure than the data warrant.