Boltzmann exploration

Epsilon-greedy treats every non-best action as equally worth a random try, which seems wasteful — surely a near-best action deserves more attention than a clearly terrible one. Boltzmann exploration fixes that by choosing actions in proportion to a softmax over their estimated values: higher-valued actions are more likely to be picked, but every action keeps a nonzero chance. The exploration is graded by how promising each option looks.

A temperature parameter sets the boldness. At high temperature the distribution is nearly uniform and the agent explores almost everything; as temperature falls toward zero it concentrates on the best action and behaves greedily, so lowering it over training mirrors simulated annealing. The catch is that it judges actions only by their estimated value, not by how uncertain that estimate is, so it can keep over-sampling an option that merely looks good while ignoring a genuinely unexplored one.