Centralized Training, Decentralized Execution

The key idea: two phases, two viewpoints

The dominant recipe for cooperative MARL is centralized training with decentralized execution (CTDE). The insight is to separate when we learn from when we act. During training — typically in a simulator — we are allowed a god's-eye view: the full joint state, everyone's observations and actions. During execution — deployment in the real world — each agent must act using only its own local observation, because that is all it will have.

CTDE directly defuses both troubles from the last guide. The centralized view at training time gives a stationary target — we condition on the joint action, so the moving-pieces problem of nonstationarity is tamed. And it gives the leverage we need to crack multi-agent credit assignment, because we can reason about everyone at once. The remaining design question is how to use that centralized signal. Three answers follow.

Value decomposition: split the team value

The first answer attacks credit assignment head-on. Value Decomposition Networks (VDN) assume the team's joint action-value can be written as a sum of per-agent values: Q_total equals Q_1 plus Q_2 and so on. Each agent keeps its own value network over its local observation; we add them up, train against the shared team reward, and gradients flow back to each agent. Because the total is a sum, an agent acting greedily on its own Q is automatically acting greedily on the team Q — execution stays decentralized.

Q_{\text{tot}}(\boldsymbol{\tau}, \mathbf{a}) = \sum_{i=1}^{n} Q_i(\tau_i, a_i)

VDN's core assumption: the team's joint action-value is the sum of per-agent values, so local greedy choices stay globally greedy.

A plain sum is restrictive, though: many teams need richer interactions than 'add up independent contributions'. QMIX relaxes the sum to any monotonic combination — the team Q rises whenever any agent's individual Q rises. A small mixing network, fed extra global state, learns this monotonic blend with non-negative weights. Monotonicity is exactly the property that keeps decentralized greedy actions consistent with the centralized optimum, while allowing far more expressive teamwork than VDN.

Counterfactual credit: what if I'd done nothing?

An elegant, policy-gradient alternative tackles the lazy-agent problem with a clever question. The counterfactual multi-agent (COMA) baseline uses a centralized critic to ask, for each agent: how much better was the team's outcome because this agent took its actual action, compared with the average over all the other actions it could have taken — holding everyone else fixed?

That difference is a counterfactual advantage: it isolates one agent's marginal contribution by subtracting a baseline that averages away the part of the reward it could not control. An idle agent gets near-zero advantage and stops being praised for the team's luck; a pivotal agent gets a large signal. It is, in spirit, the credit assignment problem solved by counterfactual reasoning.

A^{a}(s, \mathbf{u}) = Q(s, \mathbf{u}) - \sum_{u'^{a}} \pi^{a}\!\left(u'^{a} \mid \tau^{a}\right) Q\!\left(s, (\mathbf{u}^{-a}, u'^{a})\right)

The COMA counterfactual advantage: subtract a baseline that marginalizes only this agent's action while holding teammates fixed.

MAPPO: the surprisingly strong default

The methods above grew sophisticated, yet a strikingly simple recipe often matches or beats them: multi-agent PPO (MAPPO). Take PPO — the reliable single-agent workhorse — give every agent a decentralized actor on its own observation, and pair them with a centralized critic that sees the global state at training time. Pure CTDE, almost no new machinery.

# MAPPO sketch: decentralized actors, one centralized critic.
for it in range(N_ITERS):
    batch = collect_rollouts(env, actors)      # each actor uses local obs only
    # critic sees the GLOBAL state s (training-time privilege)
    values = critic(batch.global_states)
    adv = gae(batch.rewards, values)           # shared-team advantage
    for i in range(N):                          # agents may share weights
        ppo_update(actors[i], batch.obs[i], batch.acts[i], adv)
    ppo_update_critic(critic, batch.global_states, batch.returns)

Each actor conditions only on its local observation; the critic exploits the global state to stabilise learning.

The lesson echoes a recurring theme in RL: a well-tuned, well-understood baseline plus the CTDE idea goes a very long way. Reach for value decomposition or counterfactual baselines when the credit-assignment structure genuinely demands it — but try MAPPO first.