model bias and compounding error

This is the central villain of model-based RL. A learned model is never exactly right, so each prediction carries a small error. When you chain predictions into a rollout, the model's next input is its own slightly-wrong output, and the errors feed on themselves: a tiny mistake at step one becomes a bigger one at step two, until after enough steps the imagined trajectory has wandered somewhere the real world would never go.

Two failures intertwine. Model bias means the model is systematically off in some direction, and compounding error means that bias is amplified over the horizon, often growing exponentially with rollout length. Worse, a planner or policy optimiser actively hunts for high predicted reward, which lures it straight into the model's blind spots — exploiting fictitious rewards the model invented where it had no data. Mitigations include short rollouts, uncertainty-aware ensembles, branching imagined rollouts off real states, and replanning often.

Understanding this error is what separates fragile model-based agents from robust ones. The whole art is to lean on the model only where it is trustworthy and to stop before its imagination decays into fiction — keeping rollouts short enough, weighting them by confidence, and constantly re-grounding the agent in real experience so that errors are corrected faster than they can accumulate.