entropy regularization

A learning policy can fall in love too early — it finds one decent action, keeps choosing it, stops trying anything else, and never discovers something better. Entropy regularization fights this premature confidence by paying the policy a small bonus for staying uncertain. Entropy measures how spread-out a distribution is, so rewarding entropy literally pays the agent to keep its options open and keep exploring.

In practice you add a term β·H(π(·|s)) to the objective, where H is the entropy of the action distribution and β is a small coefficient. The policy gradient then pushes both toward high-return actions and toward not collapsing onto a single action. Early in training β keeps exploration alive; as learning matures you often anneal β down so the policy can finally sharpen toward the best actions.

Pushed further, entropy is not just a regulariser but the centrepiece of maximum-entropy RL, where the agent maximises return plus entropy as a unified objective — the idea behind soft actor-critic. The honest caveat is tuning: too much entropy and the agent stays vaguely random and never commits; too little and it collapses early and gets stuck.