Self-play: an opponent that grows with you
How do you train a champion when no expert opponent exists? In a competitive game, the answer is to let the agent play against itself. Self-play training pits the current policy against copies of itself, generating an opponent that is always exactly as strong as the learner. This is an automatic curriculum: as you improve, your sparring partner improves in lockstep, so the challenge is never trivial and never hopeless.
This is the engine behind the famous game-playing systems. AlphaZero mastered chess, shogi and Go from nothing but the rules, by combining self-play with search — starting from random moves and bootstrapping all the way to superhuman strength. The same idea, with AlphaGo before it, showed that self-play can discover strategies human experts had never considered.
A game tree expanding from the current position, with nodes selected and expanded by Monte-Carlo tree search.
The cycling trap, and fictitious play
Naïve self-play has a famous failure: it can forget. If you only ever train against your latest self, you may overfit to the current opponent and lose to strategies you beat long ago — the rock–paper–scissors cycle from guide 2, where each new policy counters the last but is vulnerable to an older one. You go round and round without converging toward a Nash equilibrium.
The classic cure comes from game theory: fictitious self-play. Instead of best-responding to the opponent's current policy, you best-respond to the historical average of all the policies it has used. By keeping memory of the whole past, you cannot be exploited by simply rediscovering an old counter — and under broad conditions this average-based dynamic provably drifts toward equilibrium rather than orbiting it forever.
Fictitious play: each step best-responds to the historical average policy, which the average then drifts toward.
From one rival to a whole population
Scale the idea up and you get population-based training (PBT): instead of one learner, evolve a whole population of agents in parallel. Members play against each other, periodically the weak ones copy the weights of the strong (exploit), and their hyperparameters are perturbed (explore). The population maintains diversity, which is the antidote to the brittleness of training against a single opponent.
The landmark example is AlphaStar in StarCraft II, which trained a league of agents — main agents, plus 'exploiters' whose job was to ruthlessly find and punish the weaknesses of the others. Beating the league requires being robust to a wide variety of strategies, not just one, pushing policies toward genuinely general skill rather than a narrow counter.
- Run many agents in parallel, each with its own hyperparameters.
- Have them compete and rank them by performance.
- Exploit: poor performers copy the weights of strong ones.
- Explore: perturb the copied hyperparameters and continue training.
Robustness over winning one match
The thread tying these methods together is robustness. Beating a specific opponent is easy and fragile; the real target is a policy that performs well against the whole space of plausible opponents — which, for zero-sum games, is exactly what approaching a Nash equilibrium means. Self-play, fictitious play, and populations are increasingly powerful ways to make sure the opponents you train against cover that space.
Underfit, good-fit and overfit curves contrasting fragile specialisation with robust generalisation.
These same tools reach into mixed cooperative–competitive worlds — think teams within a league — where agents must both coordinate with allies and out-strategise rivals at once. That blend of cooperation and competition, and the open-ended behaviour it produces, is where we head next.