directed vs undirected exploration

Exploration methods split into two camps. Undirected exploration injects randomness blind to what the agent knows — epsilon-greedy and Boltzmann action noise just jitter the policy, exploring everywhere with equal disregard for which states are actually unknown. Directed exploration, by contrast, uses information: it deliberately steers toward states that are novel, uncertain, or expected to be informative, spending its exploratory budget where it can actually learn something.

The difference is the difference between feasible and hopeless on hard problems. Undirected noise is simple and works fine when reward is plentiful, but its chance of stumbling through a long required sequence shrinks exponentially, so it stalls in sparse, deep tasks. Directed methods — count bonuses, UCB, curiosity, posterior sampling — pay an overhead to track uncertainty, but that bookkeeping is exactly what lets exploration scale to the environments where blind randomness never arrives.