sample efficiency of model-based RL

Sample efficiency asks how much real-world experience an agent needs to get good. It is the headline reason anyone bothers with models: real interaction can be slow, expensive, or risky — wearing out a robot, burning lab time — so an agent that learns from fewer real steps is often worth a great deal more than one that merely learns from more compute.

Model-based methods win here by squeezing extra value from each real sample. A single real transition does triple duty: it improves the value function or policy directly, it improves the model, and through the model it generates many more imagined transitions to learn from. Model-free agents, by contrast, learn only from the real transitions they collect, so they often need orders of magnitude more environment steps to reach the same skill, even if each of their updates is simpler and more stable.

The catch is that this efficiency is not free: it is borrowed against model accuracy. As the model's errors grow, the imagined samples become misleading, so the real prize is the regime of limited data, while model-free methods often catch up or surpass model-based ones once data is abundant and the model's flaws start to bite.