generalist agents

Most RL agents are specialists: one network plays one game, or controls one robot. A generalist agent is the opposite ambition — a single set of weights that can play hundreds of games, caption an image, chat, and control different robot bodies, switching task and even embodiment without retraining. The landmark proof-of-concept, often called a generalist agent, tokenized everything (pixels, text, joint torques) into one sequence and trained one transformer to predict the next token across all of it.

The technical trick is unification: cast every modality and every action space as tokens in a common stream, so wildly different tasks become one big sequence-modeling problem a large model can absorb. This lets positive transfer flow across tasks and lets one artifact be deployed everywhere. It blends RL, imitation learning, and supervised pretraining, and leans heavily on foundation-model scale.

Generalist agents are striking demonstrations but still early. They are typically jacks of all trades and masters of few, often trailing specialists on any single hard task; handling many embodiments without interference is unsolved; and it is unclear whether scaling alone yields the deep competence dedicated agents reach. They are a vivid statement of where the field hopes to go more than a finished destination.