training-serving skew

A model is only ever as good as the consistency between the world it was trained on and the world it predicts in. Training-serving skew is the quiet, corrosive gap that opens when the features a model sees at serving time differ from those it learned on, even though both were nominally computed from the same definitions. The model performs beautifully in offline evaluation and disappoints in production, and nothing in the metrics points at the cause, because the inputs themselves have shifted out from under it.

Skew has several distinct sources. There is implementation skew, where training computes a feature in Python over a warehouse while serving recomputes it in a different language with subtly different edge cases, rounding, or null handling. There is temporal skew, where training uses a value as of the label time but serving has only a stale or differently-windowed value, a special case of label leakage in reverse. And there is distribution skew, where the live input distribution simply differs from the training sample. Each degrades accuracy without any code being broken in an obvious way.

The standard defenses are to compute features once and read them from a shared feature store at both training and serving, to log served feature values and replay them to detect divergence, and to assert invariants (ranges, schemas, missing-value rates) on both sides. The goal is a single feature definition with a single execution path, so that train and serve are by construction the same computation rather than two reimplementations hoping to agree.