counterfactual multi-agent baseline (COMA)

COMA tackles team credit assignment with a single sharp question for each agent: 'How much better or worse did the team do because you, specifically, chose this action — compared with if you had done something else and everyone else had done exactly the same?' By holding the other agents' actions fixed and only varying one agent's action, it isolates that agent's individual contribution out of the shared reward. That difference is its counterfactual advantage.

Mechanically it is a CTDE actor-critic. A centralised critic learns the joint action-value given the global state. The baseline subtracted for agent i is not a plain state value but a counterfactual expectation: the critic's value averaged over all the actions agent i could have taken, weighted by its current policy, with the other agents' actions held fixed. Subtracting this from the actual joint value gives an advantage that reflects only agent i's choice — and, conveniently, the baseline does not depend on agent i's chosen action, so it does not bias the policy gradient.

The elegance is that a single centralised critic provides a tailored, individualised learning signal to every agent at once, directly answering 'was your move good given what the team did'. The cost is that it needs that centralised critic over the joint action and works most cleanly with discrete actions, where summing over an agent's alternatives is cheap. It remains a landmark for showing how counterfactual reasoning cleanly solves cooperative credit assignment.