JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

When the Model Lies: Bias and Uncertainty

Imagined experience is only as good as the model — here is how compounding error bites, and how ensembles fight back.

The core danger: compounding error

A learned model is never exact. On a single step the error may be tiny, but rollouts feed each prediction back in as the next input. Small mistakes get re-digested step after step until the imagined trajectory wanders into states the real world would never produce. This snowball is called model bias and compounding error, and it is the single biggest reason naive model-based agents fail.

In model-based RL the learned model stands in for the environment, so each step's prediction feeds back as the next input — and its errors compound around this loop.

A loop diagram of an agent acting on an environment and receiving the next state and reward back.

First defence: keep rollouts short

The simplest, most reliable fix is to limit how far you imagine. Short rollouts branched from many real states give the policy plenty of cheap practice while never straying far enough for error to explode. This is exactly why MPC re-plans every step and why modern Dyna-style methods often imagine only a handful of steps before grounding themselves in real data again.

Honest models: predicting uncertainty

A model that outputs a single confident guess hides its own ignorance. A probabilistic dynamics model instead predicts a distribution over next states — typically a mean and a variance — so it can say "the next state is probably here, but I'm unsure." That honesty matters: where the model is uncertain, the planner can hold back, and where it is confident, it can imagine boldly.

p_\theta(s_{t+1}\mid s_t,a_t)=\mathcal{N}\!\left(\mu_\theta(s_t,a_t),\ \Sigma_\theta(s_t,a_t)\right)

A probabilistic dynamics model predicts a Gaussian over the next state — a mean plus a variance that voices its own uncertainty.

It helps to split two kinds of doubt. Aleatoric uncertainty is genuine randomness in the world; epistemic uncertainty is the model's lack of data, which more experience can cure. A good model-based agent uses epistemic uncertainty both to plan cautiously and to decide what to explore next.

Ensembles to the rescue: PETS

How do you measure epistemic uncertainty cheaply? Train several models on the same data and watch where they disagree. This is the idea behind PETS — probabilistic ensembles with trajectory sampling. It combines two ingredients: an ensemble of probabilistic networks (disagreement reveals epistemic doubt) and trajectory sampling (each imagined rollout is propagated through a randomly chosen ensemble member, so plans automatically respect the spread of opinions).

Epistemic uncertainty is Bayesian doubt that shrinks as data arrives — and a PETS ensemble's disagreement is a cheap stand-in for this posterior spread.

An interactive prior distribution narrowing into a tighter posterior as evidence is added.

Paired with MPC, PETS reaches strong control performance with remarkably little real data — a vivid demonstration of model-based sample efficiency when uncertainty is handled with respect.

Putting the defences together

  1. Train an ensemble of probabilistic models so disagreement signals where the model is ignorant.
  2. Plan with short horizons so compounding error never has room to snowball.
  3. Sample rollouts across ensemble members so optimistic single-model fantasies get averaged away.
  4. Re-plan and refit often, feeding fresh real data exactly where the model was most uncertain.