phasic policy gradient (PPG)

Most actor-critic methods train the policy and the value function together, often through a shared network. That sharing creates a tension: the value function wants many optimization passes over the same data to fit accurately, but the policy must not be trained too long on stale data or it drifts off-policy and destabilizes. Phasic policy gradient resolves the conflict by separating the two into distinct phases instead of optimizing them simultaneously.

In the policy phase, PPG does a normal PPO-style update, taking only a few passes so the policy stays close to the data that produced it. In the following auxiliary phase, it trains the value function for many more epochs on the accumulated data, and adds an auxiliary objective that distills value knowledge into the shared representation while a KL term keeps the policy output from changing. This lets the value head be optimized hard for accuracy without over-optimizing the policy, improving sample efficiency and the quality of the shared features.