policy-constraint methods

The simplest defense against distributional shift: do not let the learned policy wander far from the one that made the data. Policy-constraint methods explicitly tie the new policy to the behavior policy, so it can only improve within the region the data actually supports.

Concretely they add a penalty or a hard constraint — a KL divergence, an MMD distance, or a support constraint — between the learned policy and an estimate of the behavior policy. The optimizer then trades off maximizing value against staying inside the data's comfort zone.

The tension is fundamental. Constrain too tightly and you just reproduce the behavior policy with no improvement; constrain too loosely and extrapolation error comes flooding back. Tuning this knob — and constraining the action support rather than the exact distribution — is what separates robust methods from brittle ones.