JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Why RL Hasn't Had Its GPT Moment

RL agents can beat world champions yet trip over a slightly recolored maze. Meet the two stubborn gaps — generalization and sample efficiency — that define the whole frontier.

Superhuman, and strangely brittle

By now you have seen RL do astonishing things: master Go, control fusion plasmas, fly stratospheric balloons. So it is jarring to learn that the same agent, after this triumph, often fails the moment the world shifts a little — a new wall color, a rescaled reward, a level it never trained on. The frontier of RL is not really about inventing a cleverer loss. It is about closing the gap between narrow mastery and broad competence.

The agent–environment loop at the heart of every RL system: act, receive a reward, observe the next state, repeat — the same machine that masters Go yet replays a single task a billion times.

Diagram of an agent sending an action to the environment and receiving a reward and next state in return.

Gap one — generalization

Generalization in RL asks whether a policy trained on some states, levels, or environments still performs on unseen ones. It is much harder than supervised generalization, because in RL the agent's own actions decide which data it ever sees. A small policy error early in an episode steers it into states it never practiced, and errors compound. Worse, agents are masters of shortcut learning: they latch onto a background texture or a frame counter that happens to predict reward, and that crutch shatters under any distribution shift.

  1. Train on a fixed set of levels, evaluate on a held-out set — the standard way to even measure RL generalization (procedurally generated benchmarks made this possible).
  2. Watch the train–test gap, not just the training curve: a reward of 9/10 on training and 2/10 on new levels is overfitting, not skill.
  3. Probe what the policy actually keys on — mask the background, shift colors, add distractors — to expose the shortcut.
Train a Q-learning agent on this gridworld, then picture it slightly recolored — the policy that looks solved here is exactly what fails to generalize to unseen levels.

Interactive gridworld where an agent learns to reach a goal by Q-learning.

Gap two — sample efficiency

The sample-efficiency frontier is the second wall. Famous game-playing agents consumed the equivalent of thousands of years of play. A human reaches respectable Atari scores in minutes. This matters far beyond games: in robotics, finance, or healthcare you cannot afford millions of risky trials, so the cost of experience is the binding constraint. The deep reason is that RL must discover its own training signal through exploration, and reward can be sparse and delayed — the agent often acts for a long time before any feedback tells it whether the choice was good.

Q(s,a) \leftarrow Q(s,a) + \alpha\left[\, r + \gamma \max_{a'} Q(s',a') - Q(s,a) \,\right]

The Q-learning update nudges a value by only α times the TD error each step — which is why reaching a good policy can demand millions of transitions.

Hasn't scaling fixed everything else?

The bitter lesson — that general methods riding more compute eventually beat hand-crafted cleverness — is the spiritual backdrop of this whole track. It worked spectacularly for language. But in RL, scaling is murkier: we have no internet-sized firehose of interaction data, the objective is non-stationary because the agent's own policy keeps changing the data, and clean scaling laws for RL are still being charted. More compute helps, yet it does not automatically buy generalization the way more text bought it for LLMs.

Scaling laws: loss falls smoothly as compute and data grow — the very premise the bitter lesson rests on, and the one RL keeps testing.

A log-log plot showing loss decreasing as model scale increases.

How to read the rest of this track

Each guide ahead attacks the two gaps from a different angle. Guide 2 reuses knowledge across tasks (meta-, transfer, continual, and unsupervised RL). Guide 3 asks whether RL can borrow the foundation-model recipe and scale into generalist agents in open-ended worlds. Guide 4 turns to what we optimize — reward specification, safety, constraints, risk. Guide 5 closes with causality and an honest map of what is still unsolved. Read them in order; they build a single argument.