full backup vs. sample backup

When you update a state's value you sum over what could happen next. A full backup uses the model to weigh every possible next state and reward by its probability — thorough but expensive. A sample backup uses just one transition that actually occurred — cheap and model-free, but noisy. This contrast is the dividing line between dynamic programming and sample-based learning.

DP performs full backups: it needs the transition probabilities and touches every successor, so the cost per state grows with the branching factor. Temporal-difference and Q-learning perform sample backups: they observe one transition and nudge the value toward that single target, with no model required. Full backups are exact but infeasible when the model is unknown or the branching is huge; sample backups trade variance for the ability to learn straight from raw experience.