overestimation under approximation

Q-learning's targets take a max over actions, and a max is biased upward whenever the estimates are noisy: the action that looks best is often just the one whose error happened to be most positive. Function approximation adds exactly that noise everywhere and shares it across states, so the max systematically picks up overestimates — and bootstrapping then propagates the inflated values backward through the whole value function.

If Q has zero-mean errors, the expectation of the max over actions is at least the max of the expectations (Jensen's inequality applied to the max). Each backup uses the maxed, over-optimistic target, and the next backup builds on it, so optimism compounds. Double Q-learning curbs this by using one network to choose the maximizing action and a separate one to evaluate it, decoupling selection from valuation; clipped double-Q (as in TD3 and SAC) takes the minimum of two critics.

Overestimation is usually harmful (it distorts the policy toward overrated actions and can drive divergence), though a little optimism can aid exploration. In offline RL it is the central enemy — the agent overvalues actions absent from the data — which is why conservative methods deliberately push value estimates down.