JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Why Multi-Agent Is Hard: The Moving Target

The most natural idea — just run a separate learner per agent — quietly breaks the theory. Meet nonstationarity, multi-agent credit assignment, and slippery equilibria.

The obvious first attempt: independent learners

Faced with N agents, the simplest idea is irresistible: give each one its own copy of a single-agent algorithm and let them learn side by side, each treating the others as part of the environment. Applied to Q-learning this is called independent Q-learning (IQL). It needs no new machinery, scales trivially to many agents, and — surprisingly often — works well enough in practice.

Each independent learner runs its own agent–environment loop, treating the other agents as just part of the environment.

Diagram of the reinforcement-learning loop: an agent takes an action, the environment returns a new state and reward.

# Independent Q-learning: every agent learns alone,
# treating the others as 'just part of the environment'.
for episode in range(N_EPISODES):
    obs = env.reset()                 # one local observation per agent
    done = False
    while not done:
        acts = [agent[i].act(obs[i]) for i in range(N)]
        next_obs, rewards, done = env.step(acts)
        for i in range(N):
            agent[i].update(obs[i], acts[i], rewards[i], next_obs[i])
        obs = next_obs
Each agent updates from only its own observation, action and reward — blind to the others.

It is the right place to start and a strong baseline. But it rests on an assumption that is now false, and understanding why is the key to everything that follows.

Nonstationarity: the floor keeps moving

Single-agent learning assumes a stationary environment: the same action in the same state has the same distribution of outcomes over time. That assumption is what lets value estimates converge. But in a Markov game, when an agent treats its co-learners as 'environment', that environment is changing, because the others are learning and updating their own policies. This is nonstationarity, the central headache of MARL.

Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha\bigl[r_t+\gamma\max_{a'}Q(s_{t+1},a')-Q(s_t,a_t)\bigr]

The Q-learning update converges only if outcomes are stationary — but other learning agents keep shifting that distribution.

Concretely: agent A learns a great response to agent B's current behaviour. Meanwhile B is also improving, so B's behaviour drifts — and A's carefully learned response is now aimed at a target that has already moved. Convergence guarantees that hold for a fixed environment simply do not transfer. Worse, replay buffers make it sharper: stored transitions were generated by old policies, so they describe a world that no longer exists.

Multi-agent credit assignment: who actually helped?

Single-agent RL already struggles with the credit assignment problem across time: which earlier action earned this later reward? MARL adds a second axis — across agents. When a team shares one reward and wins, which agent's choices made the difference? This is the multi-agent credit assignment problem.

Single-agent Q-learning already faces temporal credit assignment — which earlier move in the gridworld earned the later reward?

Interactive gridworld where a Q-learning agent propagates reward back to earlier states.

Picture a five-player game where the team scores. The shared reward tells everyone 'good job' — including the agent who stood around doing nothing useful, who now gets reinforced for loafing. This is the lazy agent pathology. Without a way to disentangle each agent's contribution, learning is muddy: a brilliant move and a useless one receive the same credit. Solving this cleanly is the whole motivation for the value-decomposition methods in the next guide.

Why equilibria are slippery

Even our goalpost is troublesome. A Nash equilibrium need not be unique — a game can have many, and different agents may be aiming at different ones, leading to miscoordination. It can be hard to compute. And in competitive games, best-response dynamics can cycle forever, like an endless rock–paper–scissors where each new policy beats the last but loses to the one before.

There is also a values trap: 'no one can improve alone' does not mean 'best for all'. The prisoner's dilemma has a Nash equilibrium where both players defect — stable, yet worse for both than mutual cooperation. So MARL is not just optimisation; it inherits the deep subtleties of strategic interaction. With the diagnosis in hand, the rest of this track is about cures.