model rollouts

A model rollout is a trajectory the agent dreams rather than lives: starting from some state, it picks an action, asks the model for the next state and reward, picks another action from there, and repeats — chaining the model's one-step predictions into an imagined path through the future. These synthetic trajectories are the raw material that planning and model-based learning feed on.

Rollouts can be used in two main ways. In decision-time planning you roll out many candidate action sequences and compare their predicted returns to choose what to do now. In background learning you generate batches of imagined transitions and treat them as extra training data for a value function or policy, multiplying your effective experience. A crucial design choice is rollout length: how many model steps to chain before you stop trusting the imagination.

The power of rollouts is leverage — a single real transition, folded into a learned model, can spawn endless practice. The danger is that every additional imagined step layers one prediction error on top of the last, so long rollouts drift away from reality and short ones limit how far ahead you can reason.