multi-agent credit assignment

Imagine a team of five agents that wins a single shared reward — say the round is won — but only two of them actually did anything useful. How should learning reward the contributors and not the freeloaders? That is multi-agent credit assignment: from one team-level number, figure out which agent's action actually mattered. Get it wrong and lazy agents are reinforced while helpful ones are punished by association.

The trouble is that the shared reward does not, on its own, factor into per-agent pieces. An agent sees the team do well and cannot tell whether that was because of its own choice or in spite of it. This is the multi-agent twist on the temporal credit assignment problem from single-agent RL — there it is 'which time step deserves credit', here it is also 'which agent deserves credit', and the two are tangled together.

The major cooperative methods are essentially different answers to this question. Value decomposition (VDN, QMIX) learns to split the team value into additive or monotonic per-agent contributions; counterfactual baselines (COMA) ask 'how would the team have done if this one agent had acted differently, holding the others fixed'; and difference rewards compare the global outcome with and without an agent's action. Solving credit assignment well is what separates coordinated teams from confused ones.