human-in-the-loop RL

Most RL imagines an agent learning alone from a fixed reward signal. Human-in-the-loop RL keeps a person actively involved while the agent learns — watching, correcting, demonstrating, approving, or intervening in real time. Think of a driving instructor with a second brake pedal, or a coach who occasionally grabs the controls. That ongoing guidance shapes what the agent learns far faster, and far more safely, than reward alone ever could.

The human's involvement can take many forms: giving preference comparisons, providing demonstrations to imitate, handing out scalar approval or live corrections, vetoing dangerous actions, or resetting the agent when it gets stuck. These signals are folded into learning — as a reward model, an auxiliary imitation loss, or a safety filter on actions. The defining feature is that feedback arrives interactively during training, so the human can react to the agent's current weaknesses rather than having to anticipate them up front.

The human is the bottleneck and the safety net at once. Their time is scarce and their attention drifts, so the hard problems are when to ask for help, how to make each query count, and how to stay safe while the agent is still incompetent. Done well, it is the safest way to train in the messy real world; done carelessly, it simply burns human hours for little gain.