off-policy policy gradient

Plain policy gradients are on-policy: every update must use fresh data from the very policy you are improving, which is wasteful — you throw away each batch after one step. Off-policy policy gradients let you compute the gradient for your current policy using data collected by a different policy: old experience in a replay buffer, a separate exploratory behaviour policy, or even human demonstrations. The payoff is sample efficiency.

The standard correction is importance sampling: reweight each sampled transition by the ratio of the new policy's probability to the behaviour policy's probability, π_θ(a|s) / b(a|s). This makes the gradient unbiased in principle, but the ratio is treacherous — when the two policies disagree it can explode, and the variance blows up. Real methods tame this with clipping (as in PPO), truncation (V-trace), or by working with deterministic gradients that sidestep the ratio entirely.

The honest trade-off is bias versus variance versus data reuse. Off-policy methods reuse data and can be far more sample-efficient, but the importance weights add variance and instability, and large policy gaps break them. Much of modern algorithm design — replay, clipped objectives, trust regions — is about safely stretching how off-policy you can be.