covariate shift in imitation

Covariate shift is the reason naive imitation drifts off course. The cloned policy is trained only on states the expert visited, but at run time the policy itself decides where it goes. A tiny error nudges it into a slightly off-distribution state it never saw in training; there it errs a little more, lands somewhere stranger still, and the gap snowballs. Think of a copied driver who hugs the lane fine until one small wobble puts a wheel on the shoulder — a place the instructor never demonstrated recovering from.

The damage is quadratic, not linear, in the task horizon: a per-step error rate of epsilon can produce total cost growing like epsilon times T squared over T steps, because each early mistake creates many later states the policy was never trained for. This is the central theoretical reason behavioral cloning is brittle for long-horizon control.

Fixes all share one idea: train the policy on the states it will actually visit, not only the expert's. DAgger does this by querying the expert on the learner's own rollouts; adversarial methods like GAIL match the learner's full state distribution to the expert's instead of matching actions point by point.