maximization bias

Suppose several actions truly have the same value, but you only have noisy estimates of each. If you take the maximum of those estimates, you almost always pick one that happened to be overestimated by chance — so the maximum systematically reads higher than the real best value. This upward distortion, baked into any rule that maximizes over uncertain numbers, is maximization bias.

It is exactly the trap that one-step Q-learning falls into, since its target maximizes over noisy next-state action values, inflating Q-estimates and sometimes steering the policy toward genuinely worse actions that merely looked good. The standard cure is to separate the act of choosing the best action from the act of valuing it, using two independent estimators — the idea behind double Q-learning and its deep counterparts.