extrapolation error

Ask a value network how good an action it has never seen would be, and it will answer with total confidence — and often nonsense. Extrapolation error is exactly this: function approximators happily produce values for state-action pairs absent from the data, and those values can be wildly, arbitrarily wrong, because nothing in the data ever constrained them.

It turns catastrophic through bootstrapping. Q-learning's target takes a max over actions; if some unseen action carries a spuriously high value, the max selects it, that inflated number propagates backward into other states, and the whole value function balloons. The policy then confidently chases a mirage.

This is why naive off-policy algorithms like DQN or DDPG often collapse when run purely offline. Naming extrapolation error as the culprit launched modern offline RL: every fix either avoids querying unseen actions or actively pushes their values down.