risk-sensitive RL

Standard RL maximizes expected return — the average over many runs. But an average hides the tails: a policy with a great mean might occasionally fail catastrophically, and for a self-driving car, a medical treatment, or an investment, the rare disaster is what you actually care about. Risk-sensitive RL changes the objective so the agent optimizes for risk or the worst case, not just the mean, deliberately trading a little average performance for far smaller downside.

It draws on tools from finance and decision theory. Some methods optimize a risk measure such as conditional value-at-risk, which focuses on the average of the worst few percent of outcomes; others use an exponential utility that penalizes variance, or distributional RL that learns the whole distribution of returns rather than its mean and then acts cautiously on it. The knob is a risk parameter that interpolates from risk-neutral to deeply risk-averse behavior.

This matters wherever stakes are real and outcomes are uncertain, and it overlaps with safe RL — though a risk measure softly discourages bad tails rather than imposing a hard constraint. Open issues include estimating tail risk reliably from limited data, the instability of training on rare events, and choosing a risk attitude that matches a human stakeholder's actual tolerance.