JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Mastering games: from Atari to AlphaZero self-play

Games gave RL its most famous wins. See why they are the perfect testbed, how pixel agents conquered Atari, and how self-play taught machines to beat grandmasters from scratch.

Why games are the perfect testbed

Games are catnip for RL researchers, and not by accident. RL for game playing offers a crisp reward (the score, or win/lose), unlimited cheap data (just play again), perfect reproducibility, and a clear human yardstick. A game is a self-contained Markov decision process you can run a million times overnight.

That makes games the place where new ideas get stress-tested before anyone trusts them on a robot or a power grid. Many techniques you meet elsewhere — replay buffers, target networks, search-plus-learning — were forged on games first.

The simplest game-world testbed: watch Q-learning discover a winning policy by trial and error — the clean arena where new RL ideas are first proven.

An interactive gridworld where a Q-learning agent learns action values to reach a goal by trial and error.

Conquering Atari from raw pixels

The breakthrough that announced deep RL was a single network playing many Arcade Learning Environment games straight from the screen. A Deep Q-Network looked at stacked frames of pixels and learned which joystick action maximized future score — no game-specific features, the same architecture across dozens of titles. The Atari benchmark became the shared yardstick for 'human-level' play.

Conquering Atari from raw pixels: a Deep Q-Network reads the screen through a convolutional pipeline and turns it into action values.

A convolutional network pipeline: image to convolution to pooling to features to output.

Self-play: learning with no teacher

Board games like Go and chess have astronomically large state spaces and no obvious feature hints. The leap that cracked them was self-play: the agent plays against copies of itself, so its opponent is always exactly at its own level. As it improves, the opposition improves in lockstep — an automatic curriculum that never gets too easy or too hard.

AlphaZero combined self-play with Monte Carlo Tree Search: a neural network proposes promising moves and estimates who is winning, and a search uses those guesses to look ahead. The improved search results become training targets for the network, which then guides better search — a virtuous loop. Starting from nothing but the rules, it surpassed the strongest prior programs in Go, chess, and shogi.

Monte Carlo Tree Search expanding promising lines — the search AlphaZero pairs with its neural network.

A growing search tree that focuses simulations on the most promising moves.

while training:
    game = new_game()
    while not game.over():
        move = mcts_search(net, game)   # look ahead, guided by net
        game.play(move)
    z = game.outcome()                  # +1 win / 0 draw / -1 loss
    replay.add(states, search_policies, z)
    net.train(replay)                   # learn move-probs and value
AlphaZero-style loop: search to play, label every position by the final outcome, train, repeat. The only supervision is who won.

What games can — and cannot — tell us

Games proved that RL can reach superhuman skill in vast decision spaces. But a game gives you a perfect simulator, free resets, and a reward handed to you by the rules. Most real applications offer none of those luxuries — which is exactly the gap the rest of this track is about.

Self-play and search-plus-learning still echo far beyond board games: they power agents in real-time strategy games, and the same 'generate data by playing against yourself' idea now shapes how large models are trained on reasoning. Games remain RL's loudest proof of concept.