JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Getting What We Asked For: Reward, Safety, Constraints, and Risk

Optimizers grant your literal wish. The frontier of safe RL is about specifying what we actually mean, staying inside hard limits, and respecting the tail of catastrophic outcomes.

The specification problem

An RL agent is a relentless optimizer of whatever number you hand it — and the reward specification problem is that the number we write down is almost never exactly what we want. A boat-racing agent learns to spin in circles collecting power-ups instead of finishing; a cleaning robot maximizes 'mess removed' by knocking over a vase to sweep it up. The reward was satisfied. The intent was betrayed. Designing rewards that capture true intent — without exploitable loopholes — is one of RL's deepest unsolved problems and a cornerstone of AI alignment.

RLHF tackles the specification problem by learning the reward from human preferences rather than hand-writing it — yet the learned proxy can still drift from what we mean.

RLHF pipeline: human preference comparisons train a reward model that then tunes the policy.

Reward hacking and Goodhart's law

When the agent exploits a flaw in the reward to score high while defeating its purpose, that is reward hacking. It is the RL incarnation of Goodhart's law: *when a measure becomes a target, it ceases to be a good measure.* The danger grows precisely as agents get more capable — a stronger optimizer finds subtler loopholes faster. This is acute in RLHF, where the agent optimizes a learned proxy: push too hard and you get reward-model over-optimization, where the policy's true quality falls even as its proxy score keeps climbing.

Constrained MDPs — optimize, but never cross this line

Folding every safety concern into one scalar reward via penalties is brittle: tune the penalty wrong and the agent either ignores safety or never acts. The constrained MDP (CMDP) framework separates the two. You maximize reward subject to explicit cost constraints — e.g., 'maximize speed such that expected collisions stay below ε.' That structure matches how engineers actually think about safety: an objective plus inviolable limits, rather than one tangled number.

  1. Define a reward to maximize and one or more cost signals to bound.
  2. Set a threshold for each cost (the line you refuse to cross, in expectation).
  3. Solve with a Lagrangian: a learned penalty multiplier rises automatically when a constraint is violated and relaxes when there is slack.
\max_{\pi}\;\mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty}\gamma^{t} r_t\right]\quad\text{s.t.}\quad \mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty}\gamma^{t} c_t\right]\le d

A constrained MDP: maximize expected reward subject to keeping each expected cost below its threshold d.

Risk-sensitive RL — averages hide disasters

Standard RL maximizes expected return, but an average is indifferent to how outcomes spread. A policy that is excellent on average yet occasionally catastrophic — a portfolio that usually profits but rarely blows up, a drone that mostly lands but sometimes crashes — can look optimal to the mean while being unacceptable in practice. Risk-sensitive RL optimizes a risk measure of the return distribution instead: variance penalties, or tail measures like Conditional Value-at-Risk that focus squarely on the worst-case fraction of outcomes.

\mathrm{CVaR}_{\alpha}(Z)=\mathbb{E}\!\left[\,Z \mid Z \le \mathrm{VaR}_{\alpha}(Z)\,\right]

Conditional Value-at-Risk: instead of the mean return, optimize the average over only the worst α-tail of outcomes.

Safe RL is a system, not a trick

Safe RL braids these threads together: careful specification, constraints that hold during learning (not just at convergence), risk-aware objectives, and safety even while the agent is still exploring — because a robot cannot break itself to discover that breaking is bad. Mature recipes layer defenses: constrained optimization, a conservative fallback or shield that overrides dangerous actions, uncertainty-aware exploration, and human oversight. Treat safety as a property of the whole pipeline, a core theme of AI safety — never as one loss term bolted on at the end.