Trustworthy predictions: calibration, conformal, continual, open problems

Is your 90% really 90%?

Uncertainty is only worth having if it is honest, and the first test of honesty is calibration: of all the times the model says it is 90% confident, is it right about 90% of the time? A reliability diagram bins predictions by stated confidence and plots accuracy against it; expected calibration error summarises the gap. Deep networks are notoriously *over*confident, and — a crucial point — being more accurate does nothing to fix this. Accuracy and calibration are separate axes.

\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\,\bigl|\,\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\,\bigr|

Expected Calibration Error: the gap between stated confidence and actual accuracy, averaged over confidence bins.

The good news: most BDL methods improve calibration as a side-effect of averaging — deep ensembles especially. The caution: calibration is usually measured in-distribution, and it degrades sharply under distribution shift. A model calibrated on clean test data can be wildly overconfident on corrupted or shifted inputs, which is exactly when you needed it to be humble.

Guarantees without a correct model

Everything Bayesian so far gives uncertainty that is only as trustworthy as the prior and the approximation. Conformal prediction makes a startlingly different promise: a finite-sample, distribution-free coverage guarantee. Pick a target like 90%, and conformal will output a prediction set (a set of labels, or an interval) that provably contains the truth at least 90% of the time — no assumption that your model is correct, no assumption on the data distribution beyond exchangeability.

Train any model you like — a Bayesian net, a plain classifier, anything; conformal wraps around it as a black box.
On a held-out calibration set, score each example by a nonconformity measure (e.g. 1 minus the softmax probability of the true label).
Take the empirical quantile of those scores at your target level — this threshold is all the calibration you need.
For a new input, output every label whose score falls under the threshold; the set's size is now your uncertainty signal.

\mathbb{P}\bigl(Y_{n+1}\in\mathcal{C}(X_{n+1})\bigr)\ge 1-\alpha

The conformal guarantee: the prediction set covers the true label with probability at least 1 minus alpha, for any model and any distribution.

Learning forever without forgetting

Bayes has an elegant answer to continual learning: when a new task arrives, yesterday's posterior becomes today's prior. Variational continual learning runs this recursion — fit a variational posterior on task 1, carry it forward as the prior for task 2, and so on. Weights the first task pinned down stay anchored by that prior, resisting being overwritten, which is precisely the protection against catastrophic forgetting that ordinary fine-tuning lacks.

p(\theta\mid\mathcal{D}_{1:t})\;\propto\;p(\mathcal{D}_t\mid\theta)\,p(\theta\mid\mathcal{D}_{1:t-1})

Bayesian continual learning as recursion: yesterday's posterior becomes today's prior, task after task.

The same recursion underlies popular regularisers like elastic weight consolidation, which is essentially a Laplace prior carried between tasks. The limitation is honest: the chain is only as good as each posterior approximation, and errors compound over a long sequence of tasks, so a small coreset of replayed examples is often added to keep the recursion from drifting.

Honest limitations

The prior is a fiction we rarely understand. A Gaussian on weights is chosen for convenience, not belief, and the cold posterior effect is the bill coming due for that mismatch.
OOD detection is not solved. No method reliably flags every input it should; OOD detection still fails on adversarial and near-distribution inputs.
Uncertainty has a compute price — sampling, ensembling, Hessians — and the honest question is always whether a cheaper baseline plus conformal would do.
Approximate posteriors can be confidently wrong. A tight mean-field Gaussian can report low uncertainty exactly where the true posterior is broad — false humility's evil twin.

Where the field is going

Three currents are worth watching. First, scaling uncertainty to foundation models: full BDL is out of reach for billion-parameter networks, so the action is in last-layer Laplace, LoRA-style low-rank posteriors, and cheap ensembles of adapters. Second, uncertainty for decisions, where the goal is no longer a calibrated number but a better downstream action — in active learning, experimental design, and safe deployment. Third, a slow reconciliation between Bayesian model averaging, deep ensembles, and conformal guarantees, as the community accepts that no single framework owns trustworthy prediction.