safe reinforcement learning

A standard RL agent will happily drive off a cliff a thousand times if that is how it discovers the cliff is bad — fine in a video game, unacceptable for a real car, drone, or surgical robot. Safe reinforcement learning is the study of learning good behavior while respecting safety constraints, both during training (don't cause harm while exploring) and at deployment (never violate the limits, even rarely).

Several formal stances exist. One encodes safety as hard constraints and optimizes return subject to bounded expected cost — the constrained-MDP view. Another shapes the objective to be risk-sensitive, caring about worst cases rather than the average. A third uses a safety layer or shield that filters out unsafe actions before they execute, often backed by a learned or hand-built model of which states are recoverable. Many add conservative exploration so the agent probes cautiously rather than blindly.

Safe RL is the bridge between RL research and real-world deployment, and it is far from solved. Guaranteeing a constraint holds with high probability under an imperfect model, learning safely without ever entering catastrophe even once, and trading off safety against performance are all active problems. It connects directly to constrained MDPs, risk-sensitive RL, and the wider field of AI safety.