temperature scaling / calibration

A classifier is calibrated when its confidence matches its accuracy: among all predictions it makes with 80 percent confidence, about 80 percent should be correct. Modern deep networks are typically overconfident, routinely outputting 99 percent confidence while being right far less often, which is dangerous wherever the probability itself drives a decision, such as deferring uncertain medical cases to a human. Calibration is the act of correcting this mismatch, and temperature scaling is the simplest and most popular method.

Temperature scaling divides the logits, the pre-softmax scores, by a single positive scalar T learned on a held-out validation set, then applies softmax as usual. A T greater than 1 softens the distribution, lowering overblown confidences, while a T less than 1 sharpens it. Because T is one number that scales all logits equally, it never changes which class is the argmax, so it improves the probabilities without altering accuracy at all, a rare free lunch. T is fit by minimising negative log-likelihood, or expected calibration error, on the validation set.

Calibration quality is usually summarised by the Expected Calibration Error, computed by bucketing predictions by confidence and measuring the average gap between confidence and accuracy in each bucket, and visualised with a reliability diagram. Temperature scaling is the standard first thing to try; richer alternatives such as vector or matrix scaling and Dirichlet calibration exist but risk overfitting and can change the predictions.