calibration of probabilities
/ kal-ih-BRAY-shun uv prob-uh-BIL-ih-teez /
A model that outputs probabilities is well-calibrated if its confidence matches reality over the long run. Think of a weather forecaster: on all the days they say "70 percent chance of rain," it should actually rain about 70 percent of the time. If it rains on only 40 percent of those days, the forecaster is overconfident — their numbers don't mean what they claim. Calibration is whether a stated probability can be taken at face value.
This is separate from, and not implied by, being accurate. A model can be excellent at ranking which cases are more likely than others — high AUC, great accuracy — while its actual probability numbers are badly off, systematically too high or too low. Modern neural networks are notorious for being overconfident: they'll happily report 99 percent certainty on answers that are right only 80 percent of the time. To check calibration you group predictions by their stated confidence and see whether each group's real success rate matches.
Why it matters enormously: the moment a probability feeds a decision, its honesty becomes the whole point. A doctor weighing a treatment, a lender setting a rate, a self-driving car judging a risk — all need the number to mean what it says, not just to rank things correctly. The reassuring news is that calibration can often be repaired after training, with simple techniques that gently stretch or squeeze the outputs (like Platt scaling or temperature scaling) without retraining the whole model. A confident-sounding model is not the same as a trustworthy one, and calibration is how you tell them apart.
Collect every case where a model said "80% sure." If it was actually right on about 80% of them, it's calibrated. If it was right on only 60%, it's overconfident — and a doctor or loan officer trusting that "80%" at face value would be systematically misled, even if the model's accuracy looks fine overall.
Calibration check: does "80% sure" actually come true 80% of the time?
Calibration and accuracy are independent: a model can be accurate yet badly calibrated, or well-calibrated yet inaccurate. Don't read a confidence score as truth without checking calibration first — and remember that calibration measured on one population can break when the model meets data from a different distribution.