V-trace off-policy correction

Large-scale RL often runs hundreds of actors generating experience in parallel while a central learner updates the policy. By the time a trajectory reaches the learner, the policy has already moved on, so the data is slightly off-policy — collected by a behaviour policy that lags the current target policy. V-trace, introduced with the IMPALA architecture, is the correction that lets the learner still compute correct value targets from this stale, distributed experience without throwing it away.

It works by building bootstrapped value targets from temporal-difference errors, each weighted by a clipped importance ratio. Two separate clipping levels do two jobs: one ratio, capped low, controls how far the correction propagates back through the trajectory and keeps variance bounded; another, the per-step ratio, sets the fixed point the value function converges toward. Clipping trades a controlled bias for stable, low-variance learning across many lagging actors, making V-trace a backbone of high-throughput distributed RL systems.