model-based vs model-free RL
/ MOD-ul-bayst vurs MOD-ul-free /
This split is about whether the agent bothers to build a mental model of how its world works. A model-free agent never tries to understand the rules — it just learns, by raw trial and error, which actions tend to pay off, like a person who gets good at a video game purely by playing it thousands of times without ever reading the manual. A model-based agent instead learns a predictive model — 'if I do this, that will probably happen' — and can then plan by imagining outcomes in its head before acting.
The trade-off is sharp. Model-based agents can be vastly more sample-efficient: once you have a decent model, you can rehearse millions of scenarios internally without touching the real, expensive world — a huge win when each real trial costs time, money, or a broken robot. The catch is that the model is never perfect, and an agent that plans against a flawed model will confidently chase fantasies, exploiting the model's errors rather than reality.
Model-free methods (Q-learning, DQN, PPO) dominated the famous early wins precisely because they sidestep the brutal difficulty of learning an accurate world model — they're simpler and robust, just hungry for experience. Model-based methods are powerful when experience is scarce or a reliable simulator exists, and modern systems increasingly blend the two: AlphaZero, for instance, plans with a known game model, while MuZero learns its own model and plans inside it. There is no settled winner; it depends entirely on how costly real experience is and how learnable the world's rules are.
Learning to drive: a model-free learner only memorizes 'in this exact situation, brake' from countless trials. A model-based learner builds a little physics model — 'at this speed on wet road, stopping takes this far' — and can then plan for situations it has never actually been in, by simulating them.
Model-free: learn what works by doing. Model-based: learn how the world works, then plan in imagination.
Model-based RL's efficiency comes with a hidden hazard: the agent plans against its learned model, so any error in that model becomes a flaw the planner can exploit, confidently optimizing for things that won't actually happen. A model good enough to plan with is itself hard to learn — which is why model-free methods, for all their data hunger, remained the safer default for years.