competitive multi-agent RL

Competitive MARL is the world of contests: my gain is your loss. The cleanest case is a two-player zero-sum game like chess, Go, or poker, where the rewards of the two sides add up to zero — whatever one wins, the other loses by the same amount. There is no point coordinating; the entire task is to find a strategy that holds up against an opponent who is trying just as hard to beat you.

Mathematically this is a Markov game with opposing reward functions, and the natural objective is minimax: play to maximise your own return assuming the opponent plays to minimise it. The relevant solution is a minimax (saddle-point) equilibrium, which in two-player zero-sum games coincides with the Nash equilibrium and, crucially, is guaranteed to exist. That guarantee is why competitive settings are theoretically cleaner than general-sum ones.

The practical engine of competitive MARL is self-play: an agent improves by playing against copies or past versions of itself, ratcheting up the difficulty as it gets stronger. This is how AlphaGo, AlphaZero, and the poker and StarCraft systems reached superhuman play. The risk to watch for is overfitting to your own quirks — beating yesterday's self is not the same as beating a genuinely different opponent.