model-based policy optimization (MBPO)

Model-based policy optimization learns a model of how the world responds and then practices inside that imagined simulator instead of relying on real experience alone. The crucial discipline is to trust the model only for short imagined rollouts, because dynamics errors compound the further you simulate.

It trains an ensemble of probabilistic dynamics models that captures both the noise in the environment and the model's own uncertainty about regions it has not seen. From real states drawn out of the replay buffer it branches many short rollouts, and an off-policy actor-critic such as soft actor-critic is trained on the mixture of real and model-generated transitions. The short-horizon, ensemble design comes with a monotonic-improvement bound: as long as the model error and rollout length stay small, optimizing in the model improves the real policy.

Short rollouts keep model bias bounded, and the ensemble discourages the policy from exploiting a single model's quirks. The result is markedly better sample efficiency than purely model-free learning.