Double DQN

Standard DQN has a systematic flaw: the max in its target both picks the best next action and reads off its value from the same noisy network. Whenever the estimates contain random error — and they always do — taking the max preferentially picks actions whose value happened to be overestimated, so the target is biased upward. Errors compound, values inflate, and the agent becomes overconfident. Double DQN removes this overoptimism with a small change.

The fix is to decouple the two jobs. Use the online network to choose which action looks best in the next state, but use the separate target network to evaluate that chosen action's value. Because the action is selected and scored by two different networks, an action overrated by one is unlikely to be overrated by the other, and the upward bias largely cancels. It costs essentially nothing — the target network already exists — yet measurably improves both value accuracy and final scores across the Atari suite.