independent Q-learning

Independent Q-learning is the simplest possible thing to try in a multi-agent world: give each agent its own ordinary single-agent learner, and let it pretend everyone else is just part of the environment. Each agent runs plain Q-learning on its own observations, actions, and rewards, blissfully ignoring that the 'environment' includes other minds that are also changing. It is the natural first baseline, and it is surprisingly often a decent one.

Mechanically, agent i keeps its own action-value function over its own local observation and action and updates it with the usual temporal-difference rule. Nothing in the update acknowledges the other agents. This makes IQL trivial to implement and perfectly scalable in the number of agents, since each just runs its own copy, and it sidesteps the exponential joint action space entirely.

The price for that simplicity is non-stationarity. Because the other agents are learning too, the effective dynamics each agent faces keep changing, which violates the stationarity that Q-learning's convergence proof relies on — so the values can chase a moving target and oscillate or diverge. Despite this, IQL with deep networks remains a respected baseline and sometimes a strong one, which is a reminder that the theory's worst case is not always the practical case.