self-play training

Self-play is the deceptively simple idea behind some of RL's most famous triumphs: to get good at a game, play against yourself. Instead of needing a library of human opponents, an agent generates its own training opponents by playing copies or earlier versions of itself. As it gets stronger, so do its opponents, so the challenge automatically scales to stay just hard enough — a built-in curriculum that lifts itself by its own bootstraps.

The mechanism is a loop: the current policy plays games against itself (or a pool of its past checkpoints), those games become training data, the policy improves, and the new, stronger policy becomes the next opponent. In two-player zero-sum games this process is naturally pulled toward the minimax equilibrium, and combined with search and deep networks it produced AlphaGo, AlphaZero, and superhuman play in poker, Dota, and StarCraft.

Its famous failure mode is cycling and overfitting to oneself. An agent can learn to beat its current self while quietly forgetting how to handle older strategies, leading to rock-paper-scissors-like cycles where progress goes in circles rather than upward. The standard guards are to keep playing a diverse pool of past opponents and to mix in averaged strategies — which is exactly what fictitious self-play and population-based methods are for.