Safe Release: Shadow, Canary, Champion–Challenger, Rollback

The release problem in ML

Software deployment asks 'does the new version still pass the tests?' ML deployment asks a harder question: 'does the new model behave better on live traffic I have never seen?' Offline metrics on a held-out set are necessary but never sufficient — the test set is the past, and production is the present. A model that wins on validation can lose in production because of distribution shift, latency regressions, or feedback effects the offline evaluation could not see.

So we never swap models; we graduate them along a ladder of increasing exposure, each rung answering a different risk before granting more traffic. The rungs are shadow, canary, and champion–challenger, with rollback as the safety net under all of them. Every rung references models by registry id and is served by the same model serving layer, so promotion is a config change, not a redeploy.

Rung 1 — Shadow deployment

Shadow deployment runs the new model on real production traffic but throws its outputs away — users still receive the incumbent model's predictions. The new model sees exactly what production sees; you compare its outputs to the incumbent's offline, with zero user risk. Shadowing answers the questions offline evaluation cannot: does the new model survive real input distributions, real latency budgets, real malformed requests, real feature freshness?

Shadowing is also where training–serving skew finally surfaces in the open: because the shadow model computes features through the live serving path, a divergence from its offline behavior is skew, caught before a single user is affected. The cost is real — you pay to run inference twice — but it is the cheapest possible insurance against a confidently-wrong launch.

\mathrm{PSI}=\sum_{k}\left(p^{\mathrm{serve}}_{k}-p^{\mathrm{offline}}_{k}\right)\ln\!\frac{p^{\mathrm{serve}}_{k}}{p^{\mathrm{offline}}_{k}}

Training–serving skew made measurable: the population stability index sums the log-ratio drift between the shadow model's live and offline feature distributions.

Rung 2 — Canary deployment

Once shadowing proves the machinery is sound, canary deployment serves the new model's outputs to a small slice of live traffic — 1%, then 5%, then 25% — while watching guardrail metrics on that slice. If latency, error rate, or business KPIs on the canary stay healthy, you widen the slice; if any guardrail trips, you route the slice back to the incumbent instantly. The name comes from the canary in a coal mine: the small exposed population gives early warning before the whole mine is at risk.

Canarying is naturally paired with A/B testing: the canary slice and the incumbent slice form a controlled experiment, and you read the difference with the statistical rigor from Volume I rather than eyeballing a dashboard. The discipline is to pre-register your guardrails and your decision rule — what metrics, what thresholds, what minimum sample — before you start, so you cannot rationalize a bad launch after the fact.

z=\frac{\hat{p}_{c}-\hat{p}_{i}}{\sqrt{\hat{p}\,(1-\hat{p})\left(\dfrac{1}{n_{c}}+\dfrac{1}{n_{i}}\right)}}

Reading the canary as a controlled experiment: a two-proportion z-test on candidate versus incumbent decides whether the KPI difference is real signal or noise.

Pin guardrails (latency p99, error rate, primary KPI) and a minimum sample before launch.
Route 1% of traffic to the candidate; hold until the sample bar is met.
If all guardrails healthy and the KPI lift is significant, widen to 5% → 25% → 100%.
If any guardrail trips at any width, auto-route the slice back to the incumbent.

Rung 3 — Champion–challenger

Canarying is an event; champion–challenger is a standing institution. You designate the incumbent as the champion and continuously run one or more challengers against it on production traffic, comparing them on the same live metrics. A challenger that beats the champion by a pre-agreed margin, sustained over enough data, is promoted to champion — and the cycle continues forever. It turns model improvement from a series of risky big-bang launches into a steady, evidence-driven competition.

The framework forces healthy questions to the surface: what is the margin a challenger must clear (so you do not promote on noise)? How do you avoid multiple-comparisons inflation when many challengers compete at once? Champion–challenger is where the safe-release ladder meets the continuous-training loop from the last guide: each auto-retrained model enters as a challenger, never as an automatic champion.

\hat{\Delta}-z_{1-\alpha/m}\,\mathrm{SE}(\hat{\Delta})>\delta

Promote only on a margin you can trust: the lower confidence bound of the improvement, Bonferroni-corrected across m challengers, must clear the margin δ.

The safety net — rollback

Every rung above assumes one thing: that when something goes wrong, you can get back to safety fast. Rollback strategies make reverting to a known-good model a single, fast, low-risk operation. Because models are immutable registry nodes, rollback is not a rebuild — it is re-pointing the serving config at the previous champion's id. The target is a rollback measured in seconds, executable by an on-call engineer at 3 a.m. without a deploy pipeline.

Shadow proves the machinery; canary limits the blast radius; champion–challenger institutionalizes evidence-based promotion; rollback guarantees a fast exit. Together they let you ship boldly because you can retreat instantly. But every rung depends on trustworthy live metrics — and computing those is monitoring, the subject of the next guide.