distribution shift
A student who memorized last year's exam may ace a re-run of it and then flounder when this year's questions are worded differently, even though the subject is the same. Machine-learning models face the same trap. They are trained on data drawn from one set of conditions, then used in the real world, where conditions are never quite identical. Distribution shift is the name for this mismatch: when the situations a model meets after deployment differ from the ones it was trained on.
'Distribution' refers to the statistical pattern of the data — what kinds of inputs are common, and what answers go with them. A medical model trained on images from one hospital's scanners may misbehave on another hospital's machines; a chatbot trained mostly on formal text may stumble on slang; a model trained before some world event has never seen it. Inputs that fall outside the training pattern are called out-of-distribution (OOD), and a model's behavior on them is much less predictable than its tidy test-set scores suggest.
Distribution shift is central to safety because the world always shifts, and adversaries shift it on purpose (adversarial examples and jailbreaks are extreme, deliberate distribution shifts). It is also where two failure modes live: a model may simply lose competence out of distribution, or — more worryingly — keep acting competently while pursuing the wrong goal, a pattern called goal misgeneralization. Either way, strong performance on data resembling training is no promise of good behavior once the situation drifts.
A vision system trained to spot pedestrians in daylight footage performs beautifully in testing, then fails at dusk and in heavy rain — conditions under-represented in its training data.
A model is only as trustworthy as its training data is representative; the world rarely stays inside the training distribution.
Out-of-distribution behavior is hard to predict from in-distribution test scores. Distribution shift can cause a model either to lose competence or to keep acting capably toward the wrong objective (goal misgeneralization) — the second is the more dangerous case.