baseline (in policy gradients)

A baseline answers a simple grievance with REINFORCE: if every reward in a game is positive, the algorithm makes every action more likely and only sorts out which ones are best painfully slowly. The fix is to judge each action against a reference point — was this return better or worse than what you'd normally expect from this state? Subtract that expectation, and only above-average actions get reinforced.

The key fact is that you can subtract any baseline b(s) that does not depend on the action without changing the gradient's expected value, because the expected score under the policy is zero. So the gradient becomes ∇_θ log π_θ(a|s) times (G_t − b(s)). The bias is untouched; only the variance changes. Choosing b(s) close to the state's value V(s) shrinks the weights toward zero on average, which is exactly what you want.

The best practical baseline is a learned estimate of V(s) — and once you train such an estimate alongside the policy, you have crossed into actor-critic territory, where (G_t − V(s)) becomes an advantage. A baseline is the single highest-leverage variance reduction in policy gradients, and the natural first thing to add to plain REINFORCE.