stochastic variational inference (SVI)

Variational inference turns posterior inference into optimization: you pick a tractable family of distributions and tune it until it best matches the true posterior. But the objective, the evidence lower bound (ELBO), classically requires a sweep over the entire dataset for every gradient step, which is hopeless when you have millions of points. Stochastic variational inference does for inference what SGD does for ordinary training: it estimates the ELBO gradient from a small random minibatch, rescales it to stand in for the full sum, and takes a noisy step.

Formally the ELBO over i.i.d. data factorizes into a sum of per-datapoint likelihood terms plus a global KL to the prior. SVI subsamples a minibatch, multiplies its likelihood contribution by N over the batch size to get an unbiased estimate of the full-data gradient, and ascends with a Robbins-Monro step schedule. Hoffman and colleagues originally used natural-gradient updates on the global variational parameters for conjugate models; modern black-box variants combine minibatching with the reparameterization trick so the same idea applies to non-conjugate deep models.

SVI is what makes a Bayesian treatment of large neural networks even thinkable; it is the workhorse behind variational autoencoders and most weight-space variational Bayesian neural nets. The cost is variance: minibatch noise plus Monte-Carlo sampling of the expectation can make the gradient very noisy, so control-variate and reparameterization techniques matter a great deal in practice.