hard-exploration problems

Some environments are hard not because the control is delicate but because finding any reward at all demands a long, specific sequence of correct actions taken on faith. Pick up a key, cross a room, avoid a trap, and only then open a door that finally pays off — get any step wrong and you see nothing. These are the hard-exploration problems, and the Atari game Montezuma's Revenge became their notorious benchmark.

They are brutal because reward is both sparse and deep: undirected exploration would need an astronomically lucky run to chain the right actions, and even directed curiosity can stall if it forgets earlier progress. Cracking them has driven much of the field's invention — count and pseudo-count bonuses, random network distillation, Go-Explore, and demonstrations — and remains a sharp test of whether an exploration method is genuinely directed and persistent rather than merely random.