Why PPO is not the last word
PPO is on-policy: it learns only from data the current policy produced, then throws it away. That is wasteful when you run hundreds of distributed actors, because by the time their experience reaches the learner the policy has already moved on, making the data slightly off-policy. PPO also splits its compute awkwardly between policy and value learning. The frontier of this track is about fixing exactly these inefficiencies.
Diagram of the reinforcement-learning loop: an agent takes an action, the environment returns a new state and reward.
V-trace: off-policy correction at scale
Large-scale actor-learner systems generate experience faster than any single learner can consume it on-policy. V-trace is the off-policy correction that makes this safe. It reweights bootstrapped value targets with truncated importance-sampling ratios — capping them so a few wildly off-policy samples cannot blow up the estimate. The truncation trades a controlled amount of bias for a large variance reduction, letting a learner consume slightly stale data from many actors without diverging.
V-trace truncates the importance weights between the learner policy π and the behavior policy μ: ρ̄ caps what the value function converges to, while c̄ bounds the variance of the off-policy correction.
Phasic policy gradient
Sharing one network between the policy and the value function saves parameters but creates a tug-of-war: the value loss can distort features the policy needs, and you cannot train the value head harder without over-fitting the policy to old data. Phasic policy gradient (PPG) resolves this by separating training into phases. A policy phase runs ordinary PPO updates; then a periodic auxiliary phase distills value knowledge into the shared representation with many epochs, while a KL term anchors the policy so the extra value training does not move it.
PPG is a clean example of the modern mindset: keep PPO's stable core, but redesign the training schedule around it to use each sample more times without breaking the trust-region intuition.
One template to rule them all
Step back and the whole track collapses into the mirror-descent template from Guide 2: every method maximizes a surrogate while penalizing or constraining movement away from the old policy. The natural gradient picks the metric; TRPO makes the constraint hard; PPO-clip approximates it with a min; PPO-penalty and V-trace tune the divergence weight. Seeing them as one family is what lets you invent the next variant instead of memorizing the last one.
The mirror-descent template: every method here maximizes a surrogate advantage while a KL penalty keeps the new policy close to the old one.
PPO in the wild: aligning language models
The biggest surprise in PPO's career is that it became the optimizer behind reinforcement learning from human feedback. In the PPO-for-RLHF pipeline, the language model is the policy, a learned reward model scores its outputs, and PPO maximizes that reward — with one crucial addition: a KL penalty to the reference policy (the original supervised model). That penalty is the PPO-penalty / mirror-descent term from Guide 4, reappearing to stop the model from drifting into degenerate text the reward model happens to overrate.
RLHF pipeline: human preference comparisons feed a reward model, which provides the signal PPO uses to tune the policy.
This is also where the trust-region instinct earns its keep beyond control: without the KL leash the policy quickly finds reward-model over-optimization — text that games the proxy reward while getting worse by human judgment. Everything you learned about keeping updates close is, in the end, the same defense against the same failure: trusting a surrogate too far.