non-stationarity in multi-agent RL

Single-agent RL leans on a quiet assumption: the environment's rules stay put while you learn. In multi-agent RL that assumption breaks, and the break is the field's signature difficulty. From any one agent's point of view the other agents are part of the environment — and they are learning too, so the environment is literally rewriting its own behaviour as you go. You are aiming at a target that moves every time you, or anyone else, takes a step.

The concrete damage is that the transition and reward dynamics an agent experiences depend on the others' current policies, which change over time. That violates the stationarity (the Markov property over a fixed environment) underpinning the convergence guarantees of Q-learning and most single-agent methods. Experience collected a thousand steps ago may describe a world that no longer exists, so naive replay can actively mislead, and learning can oscillate or diverge.

There is no free cure, only mitigations, and naming them maps much of the field. Centralised critics condition on the other agents' actions so the signal is stationary given that extra information; opponent modelling tries to predict how others will change; population and self-play schedules slow the churn; and importance-weighting can down-rank stale off-policy data. Recognising non-stationarity as the root cause is half of understanding why MARL is built the way it is.