Robbins–Monro conditions

These are the two rules a sequence of step sizes must obey for stochastic approximation to converge. Intuitively the steps must be large enough, in total, to travel any distance the algorithm might need — so their sum diverges — yet small enough, eventually, to damp out the random noise — so the sum of their squares converges. Step sizes that stay too large never settle; step sizes that shrink too fast stall before reaching the target.

The canonical schedule α_t = 1/t satisfies both: the harmonic series diverges while the sum of 1/t² converges, and more generally α_t = c/t^p works for any exponent p in (1/2, 1]. In practice many deep-RL systems deliberately violate these conditions, using small constant step sizes, which trades asymptotic convergence for the ability to track a slowly moving target and for sheer stability on non-stationary problems.