double Q-learning

Ordinary Q-learning has a habit of overestimating action values, because the same noisy numbers are used both to pick the best next action and to judge how good it is — and whichever action got lucky upward tends to win. Double Q-learning breaks this self-reinforcing loop by keeping two separate value tables and letting them cross-check each other.

On each update it randomly picks one table to choose the action it believes is best, then reads that action's value from the other table to form the target. Because the table that selects is different from the table that evaluates, the upward luck in one no longer feeds its own estimate, and the systematic overestimation largely cancels. This decoupling is the core idea later carried into Double DQN for deep reinforcement learning.