constrained MDP

A constrained Markov decision process is the cleanest formal home for safety in RL. It keeps everything from an ordinary MDP — states, actions, transitions, a reward to maximize — and adds one or more cost signals, each with a budget. The agent's job changes from simply maximizing return to maximizing return subject to keeping the expected cumulative cost below its limit, for example: drive as fast as you can while keeping the expected number of collisions under a threshold.

Mathematically it is a constrained optimization, and the standard tool is the Lagrangian: introduce a multiplier that prices each unit of constraint violation, then jointly learn the policy and adjust the multiplier so the constraint is met at the optimum. Algorithms like constrained policy optimization and Lagrangian variants of actor-critic methods solve it in practice. Unlike a hand-tuned penalty, the multiplier is learned, so the trade-off self-calibrates to actually satisfy the budget.