tabular Q-learning

Tabular Q-learning keeps a plain lookup table with one cell for every state-action pair, holding an estimate of that pair's long-run value. The agent explores however it likes, and after each step it updates the cell it just used: it nudges the old value toward the reward received plus the discounted value of the best action available in the new state. Over time the table converges to the optimal action-values, and acting greedily on it gives the optimal policy.

The defining feature is the max in the target. Because the update assumes the agent will act optimally from the next state onward, it learns about the optimal policy even while the agent is busy exploring with, say, an epsilon-greedy rule — this is what makes Q-learning off-policy. With every state-action pair visited often enough and a properly decaying step size, it provably converges. The word tabular flags its limit: one cell per pair only works when states and actions are few.