multi-agent PPO (MAPPO)

MAPPO is simply proximal policy optimisation — the workhorse single-agent policy-gradient method — carried over to cooperative teams in the CTDE style. Each agent has its own policy that acts on its local observation, and they share (or each keep) a centralised critic that sees the global state during training. You collect joint experience, estimate advantages with that centralised value, and update each policy with PPO's familiar clipped objective. It is almost boringly straightforward, which is the point.

What made MAPPO notable is empirical, not theoretical: a careful study found that this plain on-policy recipe matches or beats far more elaborate value-decomposition methods on standard cooperative benchmarks, provided a handful of implementation details are right — a centralised value function, sensible input features (including agent id), value normalisation, and enough parallel data. It quietly overturned the assumption that cooperative MARL needs specialised off-policy machinery.

The trade-offs are the usual on-policy ones, amplified by having many agents: PPO is sample-hungry because it discards data after each update, so MAPPO leans heavily on cheap, massively parallel simulation. Where you have that, it is a strong, simple, and remarkably reliable default; where samples are precious, value-decomposition or off-policy methods may still win. Its rise is a good reminder that a well-tuned simple method often beats a fancy one.