SARSA
/ SAR-suh /
SARSA is Q-learning's more cautious twin. They share almost everything — both keep a notebook of Q-values and nudge them toward 'reward now plus value later' — but they differ in one telling way: when SARSA updates a Q-value, it uses the action it actually took next, including any exploratory blunder, rather than the best action it could have taken. It learns the value of how it really behaves, warts and all.
The name is just the five things one update needs, in order: State, Action, Reward, next State, next Action — S, A, R, S, A. That second 'A' is the whole point. Because SARSA learns from its own real next move, it is called 'on-policy': it is improving the very same exploring policy it is following, not some idealized greedy strategy it isn't actually using.
This makes a real practical difference near danger. Picture a clifftop path. Q-learning, assuming it will always play optimally, learns to hug the cliff edge for the shortest route — but a still-exploring agent that hugs the edge occasionally takes a random step off it. SARSA, which factors in those occasional random steps, learns to keep a safer margin from the cliff. Neither is simply 'better': Q-learning finds the optimal path for a flawless future actor, while SARSA finds the smarter path for an agent that knows it will sometimes slip.
The classic 'cliff walking' grid: a row of cliff squares gives a big penalty. Q-learning learns the daring path right along the cliff edge (optimal if you never slip). SARSA, still exploring with epsilon-greedy, learns the safer path one row back — because it accounts for the occasional random misstep that would send it over the edge.
Same maze, two attitudes: Q-learning plans for a perfect actor; SARSA plans for one that still explores and slips.
The single difference between SARSA and Q-learning — actual next action vs best next action — is the textbook illustration of on-policy vs off-policy. SARSA's safer, more conservative behavior near hazards isn't a flaw; it's the honest consequence of learning the value of the policy you're really running.