JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Why Copying Breaks Down: Covariate Shift and DAgger

A behaviorally-cloned policy lives in a world its training data never saw. We diagnose the compounding-error trap and fix it with DAgger.

The compounding-error trap

Supervised learning assumes train and test data come from the same distribution. Behavioral cloning quietly violates this. During training, states are drawn from the expert's behavior. During deployment, states are drawn from the learner's own behavior — and the learner is imperfect, so it visits states the expert never would.

Each small action error nudges the agent slightly off the expert's path. That puts it in a slightly unfamiliar state, where its prediction is slightly worse, which nudges it further off, and so on. In sequential decision problems a per-step error rate ε can grow the total expected mistake to roughly ε·T² over a horizon of T steps — quadratic, not linear. A policy that looks 99% accurate on held-out frames can still drive straight off the road.

Unlike supervised learning, the agent acts back into the environment, so each action error reshapes the next state — the closed loop that makes cloning errors compound.

Diagram of the agent–environment reinforcement-learning loop: the agent takes an action, the environment returns a new state and reward, and the cycle repeats.

Covariate shift, named

The precise name for this is covariate shift: the input distribution (the states, the covariates) differs between training and test, while the conditional 'right action given a state' stays the same. Behavioral cloning optimizes the wrong expectation — average loss under the expert's state distribution — when what we actually care about is loss under the learner's state distribution.

d_{\pi_\theta}(s)\;\neq\;d_{\pi^{*}}(s),\qquad \pi^{*}(a\mid s)\ \text{unchanged}

Covariate shift: the learner visits a different state distribution than the expert, while the right action for each state stays the same.

Seen this way, the fix is obvious in principle: get training data from the learner's state distribution. But there is a chicken-and-egg snag — we need expert labels for the states the learner visits, and the expert only naturally visits its own states. We need a way to ask the expert, 'what would you do here, in this state I stumbled into?'

DAgger: keep asking the expert

DAgger (Dataset Aggregation) closes the loop. Run the current learner to generate trajectories, then for every state it visited, query the expert for the action it would have taken. Add these freshly-labeled states to a growing dataset and retrain. Iterate. Because the new states come from the learner, you are finally training on the distribution you will actually be tested on.

  1. Start with a policy (e.g. one behavioral-cloning pass) and an aggregated dataset D = the original demonstrations.
  2. Roll out the current policy in the environment to collect the states it actually visits.
  3. For each visited state, ask the expert for the correct action; add these (state, expert-action) pairs to D.
  4. Retrain the policy on the whole aggregated D, then repeat from step 2.
D = expert_demos                       # initial aggregated dataset
policy = train_bc(D)
for i in range(num_iterations):
    states = rollout(policy)           # learner's OWN state distribution
    labels = [expert.action(s) for s in states]   # query the expert here
    D = D + list(zip(states, labels)) # AGGREGATE, never discard
    policy = train_bc(D)               # refit on everything seen so far
DAgger interleaves rollouts with expert queries; the dataset only ever grows, hence 'aggregation'.

Costs and caveats

DAgger trades data for interaction with an expert at training time. That is cheap when the expert is an algorithm (e.g. a slow planner you want to distill into a fast reactive policy) but expensive and awkward when the expert is a human, who must label out-of-context states the learner blundered into — sometimes dangerous or nonsensical ones.

DAgger also inherits BC's ceiling: it can only match the expert it queries, and a noisy or suboptimal expert caps performance. Still, DAgger is the canonical answer to covariate shift and a building block for richer schemes that mix imitation with reward, which we will see in demonstration-guided RL.

\text{BC: }\;J(\pi^{*})-J(\pi)\;\lesssim\;\epsilon\,T^{2}\qquad\text{DAgger: }\;J(\pi^{*})-J(\pi)\;\lesssim\;\epsilon\,T

Why DAgger wins: behavioral cloning's gap to the expert can grow quadratically with the horizon T, while DAgger keeps it linear.