fictitious self-play

Fictitious self-play fixes a specific weakness of plain self-play: chasing only your latest self can send you spinning in circles, like an agent that learns to beat scissors, then loses to a returning rock. The cure is to stop best-responding to the single most recent opponent and instead best-respond to the average of all the opponents you have ever faced. By aiming at the historical mixture rather than the moving tip, learning is pulled steadily toward equilibrium instead of cycling around it.

It descends from fictitious play, a classic game-theory procedure: each player assumes the opponent plays the empirical average of its past actions, then plays a best response, and the running averages are proven to converge to a Nash equilibrium in important classes of games such as two-player zero-sum. Fictitious self-play is the modern reinforcement-learning realisation — best responses are learned by RL, and the 'average strategy' is approximated by a network trained to imitate the agent's own past best-response play.

This averaging idea is the backbone of strong game-playing systems, most notably in poker, where neural fictitious self-play and its descendants reach near-equilibrium play in imperfect-information games. The cost is that maintaining and learning an average strategy adds machinery and slows things down compared with naive self-play, and convergence guarantees hold cleanly only for restricted game classes — but the stability it buys is usually worth it.