model-based vs model-free RL

When a robot learns to act by trial and error, there are two broad styles, and the difference is whether it builds a mental picture of how the world responds to its moves. A model-free robot skips the picture entirely: it just tries actions, sees whether things went well or badly, and gradually nudges its habits toward whatever earned more reward — like a child learning to ride a bike purely by wobbling, falling, and adjusting, without ever thinking about physics. A model-based robot instead first learns a little internal simulator — a model that predicts what will happen if it does this or that — and then uses that model to plan ahead and pick good moves, like a chess player imagining several moves before touching a piece.

Each style trades off the same two things: how much real-world practice it needs, and how much computing it does while acting. Model-free learning is simple and robust, but it can be terribly hungry for experience — a robot may need millions of attempts, which is fine in a fast simulator but painful and slow on real hardware that wears out and breaks. Model-based learning is usually far more sample-efficient, because every bit of experience also teaches the internal model, which can then be replayed and planned against many times over almost for free; its weakness is that if the learned model is even slightly wrong, the robot may confidently plan beautiful actions that fail in reality.

In practice robotics blends the two rather than choosing sides. A common pattern is to learn a rough world model from limited real data, use it to plan or to pre-train a policy, and then polish that policy with a bit of model-free trial and error so it stops relying on the model's mistakes. Understanding which style a system leans on tells you a lot about why it behaves the way it does: a model-free system that suddenly fails has simply never seen that situation, while a model-based one that fails was probably misled by a flaw in its imagined world.