hindsight goal relabeling

Here is a beautifully simple idea for the sparse-reward problem. Suppose you tried to reach the kitchen but ended up in the hallway. As a kitchen attempt it is a failure with zero reward — useless for learning. But hindsight relabeling says: pretend, after the fact, that the hallway was the goal all along. Now the very same trajectory is a perfect success, full of useful signal about how to reach the hallway. You learned nothing about kitchens but a great deal about getting around.

Mechanically, you store each transition in the replay buffer twice or more: once with the goal you actually intended, and again with goals you actually achieved — usually states you visited later in the same episode. Under the relabeled goals the reward becomes positive, so even a stream of nominal failures yields plenty of successful examples. This is Hindsight Experience Replay (HER), and it slots naturally on top of any off-policy, goal-conditioned algorithm.

The effect on sparse-reward, goal-reaching tasks can be the difference between learning nothing and learning fluently — robotic manipulation tasks that were essentially impossible flat became tractable with HER. The main caveats are that it needs a goal-conditioned setup and a known way to compute reward for arbitrary relabeled goals, and that biased relabeling distributions can sometimes mislead the value estimates.