conformal prediction

Most uncertainty methods give you a number you have to trust. Conformal prediction gives you a guarantee. Wrap it around any trained model and, for a chosen error rate like ten percent, it returns not a single label but a prediction set, or a regression interval, that is mathematically certain to contain the true answer at least ninety percent of the time, with no assumptions about the model or the data distribution beyond exchangeability.

The split (inductive) recipe is simple: on a held-out calibration set, score how strange each true label is under the model, using a nonconformity score such as one minus the softmax probability of the correct class; take the empirical 1 minus alpha quantile of those scores; at test time include every label whose score falls below that threshold. Exchangeability of the calibration and test points yields finite-sample, distribution-free marginal coverage of exactly 1 minus alpha for any underlying predictor.

It is post-hoc, model-agnostic, and increasingly the practical standard for deployed uncertainty. Two honest caveats: the guarantee is marginal, averaged over inputs rather than conditional on each input, so sets can be too wide on easy inputs and too narrow on hard ones, which adaptive variants like APS and RAPS help; and the exchangeability assumption breaks under distribution shift, which weighted and online conformal methods try to repair.