compatible function approximation

When you replace the true value with a learned critic, you risk the critic's errors corrupting the policy gradient. Compatible function approximation is a theoretical answer to the question: which critics are 'safe' — that is, which ones leave the policy gradient exactly unbiased even though they are only approximations? The answer is surprisingly specific and elegant.

A critic is compatible when its features are exactly the policy's own score, ∇_θ log π_θ(a|s), and it is fit by minimising squared error against the true action-values. Under those two conditions the error of the critic is orthogonal to the gradient direction, so plugging the approximate critic into the policy gradient theorem yields the same expected update as using the true Q. In effect the critic is a linear model in the score features, fitted by least squares.

This result is mostly of theoretical importance — in deep RL we rarely enforce exact compatibility — but it explains why advantage actor-critic methods work as well as they do, and it is the conceptual root of natural policy gradients. The lesson to carry: not every critic is harmless, and the structure of a critic, not just its accuracy, governs whether it biases the gradient.