divergence under function approximation

The failure mode the deadly triad produces: instead of settling down, the value estimates grow without bound, often oscillating then shooting to plus or minus infinity. You see the loss explode, predictions hit NaN, and the agent's behavior collapse. It is not slow learning — it is the algorithm actively pushing itself apart.

Bootstrapped, off-policy, approximate updates are not guaranteed to be a contraction. The combined update operator can have an effective gain greater than one, so each pass magnifies the error rather than shrinking it. Baird's star counterexample shows this on a tiny linear problem: TD weights diverge even though a perfect representation exists and the true values are zero.

Divergence is why naive deep Q-learning is unstable and why target networks, replay buffers, smaller or decayed step sizes, gradient clipping, and gradient-TD methods exist. Convergence in approximate RL is never automatic — it must be engineered or proven for the specific combination of components.