bootstrapping vs sampling

This is the axis along which the whole chapter is organized. To bootstrap is to build an update target out of your own current value estimates, as temporal-difference learning does; to sample is to build it out of full, real returns rolled out to the end, as Monte Carlo does. Almost every value-learning method is a particular mix of the two, and choosing the mix is one of the central design decisions in reinforcement learning.

The mix is a bias-variance tradeoff. Bootstrapping replaces noisy future rewards with a single estimate, which slashes variance but injects bias whenever those estimates are wrong — and it can diverge when combined with function approximation. Sampling uses only real outcomes, so it is unbiased but high in variance and forced to wait for episodes to end. The n-step returns and the lambda parameter exist precisely to let you slide smoothly between the two and find the sweet spot.