mirror descent policy optimization

Mirror descent is a generalization of gradient descent for problems where the natural way to measure distance is not plain Euclidean. Viewing policy optimization through this lens gives a clean, unifying story: each update maximizes the expected advantage minus a divergence term that penalizes moving away from the current policy, where the divergence is typically the KL between policies. Solve that small trade-off problem at every step and you get a principled trust-region update almost for free.

This perspective is illuminating because the familiar algorithms fall out as special cases. Natural policy gradient is mirror descent with the KL divergence as the geometry; TRPO is mirror descent with a hard KL constraint instead of a soft penalty; and PPO can be read as an approximate mirror-descent step. Casting them this way borrows convergence guarantees from the mature theory of mirror descent and clarifies why a KL-regularized update is the right primitive — the same regularization that stabilizes maximum-entropy RL and RLHF.