JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Toward General Agents: Foundation Models and Scale for RL

Can RL inherit the recipe that made language models general — pretrain on everything, then specialize? Foundation models, generalist agents, scaling laws, and worlds that never run out of challenges.

The LLM recipe, and why RL resists it

A foundation model is pretrained once on a vast, diverse corpus and then adapted to countless tasks. That recipe made language general. The dream of foundation models for RL is the same: pretrain a single agent on oceans of behavior so that any new control task is a quick adaptation rather than training from scratch. The obstacles are specific to RL. There is no ready-made internet of interaction data; behavior is tied to a particular body and action space; and reward is task-specific in a way that text's next-token objective is not.

Turn each state, action, and return into a token: control becomes one sequence the model can pretrain on with plain next-token prediction.

Tokenization pipeline: raw inputs become token ids and then embedding vectors.

RL as sequence modeling

The Decision Transformer made this concrete: feed a transformer the trajectory as a sequence of (return-to-go, state, action) tokens and train it to predict the next action by ordinary supervised learning. At test time you condition on a desired return and let it autoregressively generate matching actions — return-conditioned policies. There is no Bellman backup, no bootstrapping; the hard credit assignment is amortized into a big sequence model. This view is what lets RL plug into the foundation-model toolbox of scale, pretraining, and transfer.

The Decision Transformer attends across the whole trajectory — click a token to see which past states and returns shape the next action.

Interactive self-attention: selecting a token highlights the other tokens it attends to.

Pair this with the unsupervised RL phase from Guide 2 and a shape emerges: a long reward-free or log-driven pretraining stage builds broad competence, then a thin task-specific layer specializes it. That is the foundation-model template, ported to action.

Generalist agents — one network, many worlds

A generalist agent pushes the dream to its edge: a single set of weights that plays Atari, captions images, stacks blocks with a real arm, and chats — switching tasks by context, not by retraining. Tokenizing every modality into one sequence makes this possible in principle. Generalist agents are still far from human breadth and often trade peak per-task performance for coverage, but they are the clearest existence proof that the foundation-model paradigm can cross into embodied, decision-making territory.

Do scaling laws hold for RL?

For language, smooth scaling laws let you predict loss from model size, data, and compute — and that predictability is what made huge bets safe. Whether equally clean laws govern RL is genuinely open. Returns are noisier and bounded by the task; the data distribution shifts as the policy improves; and exploration can create sharp, non-power-law jumps. Early evidence suggests some power-law trends do appear when the setup is controlled (e.g., model-size vs. performance at fixed data), but RL adds extra axes — environment diversity, replay ratio, update-to-data ratio — that have no LLM analogue.

Smooth scaling laws: loss falls predictably with model size, data, and compute on a log-log plot — the predictability RL is still trying to earn.

A log-log plot of loss versus scale showing a straight-line power-law trend.

Open-ended learning — the data problem in disguise

If RL's scarce resource is diverse interaction, then the real bottleneck is the environment, not the agent. Open-ended learning confronts this head-on: build systems that keep generating new, ever-harder challenges indefinitely, so the agent never saturates. The richest version is a co-evolving curriculum — environments and agents pushing each other, much like self-play generated endless opponents for AlphaZero. Generalization then arises not from one fixed dataset but from a problem generator that never repeats itself. This reframes the path to general agents as an environment-design problem.