Planning with a Learned Model

What "planning" means here

In model-based RL, planning means using a model to compute better behaviour without acting in the real world. Given the ability to imagine outcomes, you can either feed those imagined experiences back into ordinary learning, or you can search explicitly over future action sequences and pick the best one. These two styles have names, and knowing the difference shapes every design choice you make.

Background planning: Dyna

Background planning improves the policy or value function between decisions, so that when the agent finally acts it can respond instantly. The classic recipe is the Dyna architecture: interleave real steps with imagined ones. Every real transition does double duty — it updates the value function directly and it improves the model, which then generates extra imagined transitions for even more value updates.

Q(S,A)\leftarrow Q(S,A)+\alpha\Big[R+\gamma\max_{a}Q(S',a)-Q(S,A)\Big]

Dyna applies this same one-step value update to imagined transitions, so each scrap of real data is reused many times over.

Dyna shines because it reuses scarce real data many times over. A refinement called prioritized sweeping makes it smarter still: instead of imagining random states, it focuses imagined updates where the value estimates just changed the most, so planning effort goes where it matters.

Decision-time planning: model predictive control

Decision-time planning does its thinking at the moment of acting: when the agent reaches a state, it simulates many candidate futures and chooses the first action of the best plan. The workhorse here is model predictive control (MPC), borrowed from classical control. The trick that makes MPC robust is receding horizon: plan a short way ahead, take only the first action, then re-plan from the new state with fresh information.

A simple decision-time planner

The most basic MPC planner just samples random action sequences, scores each one with model rollouts, and keeps the winner. It is crude but surprisingly effective, and it is the seed of fancier methods.

a_{t:t+H}^{\star}=\arg\max_{a_{t:t+H}}\;\mathbb{E}\!\left[\sum_{k=0}^{H-1}\gamma^{k}\,r(\hat{s}_{t+k},a_{t+k})\right]

Random-shooting MPC scores each plan by this predicted return from its model rollout, keeps the winner, then commits to only its first action.

def plan(model, state, H, N):
    best_seq, best_return = None, -inf
    for _ in range(N):                      # N candidate plans
        seq = sample_action_sequence(H)     # H actions long
        s, total = state, 0
        for a in seq:                       # roll the model forward
            s, r = model.predict(s, a)
            total += r
        if total > best_return:
            best_return, best_seq = total, seq
    return best_seq[0]                       # execute only the first action

Random-shooting MPC: imagine N plans, keep the best, act once, then re-plan next step.

Search-based planning: a glimpse of MCTS

When actions are discrete and the future branches sharply — as in board games — random shooting is wasteful. Monte Carlo tree search (MCTS) instead grows a search tree, spending more imagined rollouts on the promising branches and pruning the hopeless ones. It is a form of decision-time planning, and it underpins game-playing milestones like AlphaZero. We return to it in depth in the final guide.

Background or decision-time?

Need instant reactions (real-time control, fast games)? Lean on background planning so the heavy thinking is already done.
Have spare compute at decision time and a model you only half-trust? Decision-time planning with re-planning shrugs off model error.
Want the best of both? Many modern systems do both — background learning to shape a value function, decision-time search to refine the final move.