model calibration
/ MOD-ul kal-uh-BRAY-shun /
A model is calibrated when its stated confidence matches reality: of all the times it says "I'm 80% sure," it should be right about 80% of the time. A trustworthy weather forecaster is the perfect picture — when they say 70% chance of rain across a year of such days, it rains on roughly 70% of them. Calibration is not about being more accurate; it's about being honest regarding how accurate you are.
This is a distinct property from accuracy, and the two come apart constantly. A model can be highly accurate yet badly overconfident — saying "99% sure" on cases it gets right only 90% of the time. It can also be underconfident, hedging when it shouldn't. To check, you bucket predictions by stated confidence and compare each bucket's confidence to its actual hit rate; a reliability diagram plots this, and a perfectly calibrated model sits on the diagonal. When it doesn't, techniques like temperature scaling can nudge the numbers back toward honesty after training.
Calibration matters anywhere a person or system acts on the confidence number, not just the answer. A medical triage tool that says "95% benign" should be wrong only 5% of the time, or doctors will mis-prioritize. Modern deep networks are notorious for being overconfident — their raw probabilities, especially from the final softmax, often run far higher than their real accuracy, which is exactly why those numbers cannot be taken at face value without calibrating and checking.
Take every prediction where a model claimed "90% confident" and check how many were actually right. If it's 90%, the model is well calibrated at that level. If only 72% were right, the model is overconfident — its "90%" really means "about 72%," and any decision that trusts the literal number will be miscalibrated too.
Overconfidence in numbers: a '90%' that's really 72% misleads everyone downstream.
Calibration and accuracy are independent. A coin-flip model can be perfectly calibrated (always say 50%) yet useless; a sharp model can be wildly miscalibrated. You need both. And calibration measured on one data distribution can break entirely under distribution shift — a calibrated model is only calibrated for the world it was checked against.