sample-efficiency frontier

A child learns to ride a bike in an afternoon of wobbling; a famous RL agent needed the equivalent of tens of thousands of years of Atari play to reach human scores. Sample efficiency measures that gap — how much interaction an agent needs to reach a given level of skill — and the sample-efficiency frontier is the ongoing push to learn capable behavior from far less experience. It is arguably the single biggest obstacle to using RL in the physical world, where every sample costs real time, energy, or wear.

The reason it is hard is structural: RL learns from a scalar reward that is often sparse and delayed, it must explore to find that reward at all, and it bootstraps value estimates from its own noisy guesses. Each of those makes data go further only slowly. The main levers we have are model-based RL (learn a world model and plan inside it), off-policy reuse of past data through replay, representation learning that compresses observations, and offline pretraining on logged data before any live interaction.

It matters because data, not compute, is usually the binding constraint for real robots, healthcare, and education. The frontier has moved fast — modern methods reach Atari-human performance in a couple of hours of game time rather than weeks — but we are still orders of magnitude behind the few-shot adaptability of animals, and closing that gap is an open research program rather than a solved engineering task.