DAgger (dataset aggregation)

DAgger is the standard cure for covariate shift, and the idea is delightfully simple: let the learner drive, but keep the expert in the passenger seat correcting it. Train an initial policy by behavioral cloning, then run that policy in the environment to see where it actually goes. At every state it visits, ask the expert what they would have done there, add those new state-action labels to the dataset, retrain, and repeat.

Because each round collects labels precisely on the states the learner produces, the training distribution converges to the policy's own visited distribution — exactly the mismatch behavioral cloning ignored. After a few iterations the aggregated dataset covers the learner's mistakes and their recoveries, so the policy learns to get back on track rather than spiraling away.

The price is an interactive, queryable expert: you need someone or something able to label the right action in arbitrary states the learner stumbles into, which is cheap in simulation but can be awkward or unsafe with a human teacher in the real world.