The Big Idea: Learn a Model, Then Plan

What model-free learning leaves on the table

Most introductory reinforcement learning is model-free: the agent never tries to understand how the world works, it just slowly memorises which actions tend to pay off through raw trial and error. That works, but it is hungry — a model-free agent may need millions of interactions to master a task a human grasps in minutes. The reason is that every lesson must arrive through actually-experienced reward, one expensive step at a time.

Model-based RL asks a different question: *what if the agent first learned to predict what the world will do, and then used that prediction to figure out good behaviour without paying for every mistake in real life?* This is the same trick a chess player uses when she thinks three moves ahead instead of moving a real piece to see what happens.

What exactly is a "model"?

A learned dynamics model is just a function the agent trains from its own experience that answers: *given this state and this action, what state comes next, and what reward do I get?* Formally it approximates the environment's transition dynamics and its reward function — the two pieces that, in a Markov decision process, fully describe how the world responds.

A learned dynamics model: a function f_theta that predicts the next state and reward from the current state and action.

Two payoffs: data efficiency and planning

Once you have a model, you get cheap experience for free. Instead of acting in the real world, you can do model rollouts — imagined trajectories where the model plays the role of the environment. A policy can train on thousands of these imagined steps for every one real step, which is the heart of model-based sample efficiency. When real interaction is slow, dangerous, or costly (a robot arm, a power grid, a patient), squeezing more learning out of each real sample is the whole game.

The second payoff is planning: with a model you can search forward through possible futures and pick the action that looks best before committing. This is why model-based ideas power some of the most striking results in the field, from board-game champions to data-efficient robots.

The model-based loop, step by step

1. Collect a little real experience by acting in the environment.
2. Fit the model: train the learned dynamics model to predict next-state and reward from that experience.
3. Plan or imagine: use the model to generate rollouts and improve the policy or value estimates without touching the real world.
4. Act with the improved policy, collect fresh data, and repeat — each loop makes both the model and the behaviour better.

A first taste in pseudocode

# one outer iteration of a model-based loop
data += collect_real_steps(env, policy, n=1000)   # a little real experience
model.fit(data)                                    # learn next-state + reward

for _ in range(many):                              # cheap imagined practice
    s = sample_start_state(data)
    for t in range(H):                             # a short model rollout
        a = policy(s)
        s_next, r = model.predict(s, a)            # the model IS the env here
        policy.update(s, a, r, s_next)
        s = s_next