Go-Explore

Go-Explore is built on a blunt observation about why curious agents fail on the hardest games: they keep forgetting the promising places they have already found. Its fix is to maintain an explicit archive of interesting states the agent has reached. Each round it picks a remembered state, returns straight to it — by replaying the path or resetting the simulator — and only then explores onward from that frontier, banking any new states it discovers back into the archive.

Separating remember-and-return from explore is the key. Most exploration methods explore and learn at the same time, and the noise of learning makes them drift away from hard-won progress, a failure called derailment; Go-Explore refuses to forget, so it can build exploration on top of exploration. This let it shatter records on Montezuma's Revenge and Pitfall. A later robustification phase turns the discovered trajectories into a reliable, noise-tolerant policy.