bootstrapping (in RL)

To update your guess about a state's value, you could wait until the episode ends and use the actual total reward — or you could update it right now using your current guess about the next state. That second shortcut, leaning on your own estimates to improve your estimates, is called bootstrapping: pulling yourself up by your own bootstraps, estimates teaching estimates.

Every DP backup bootstraps — a state's new value is built from the stored values of its successors, not from observed final outcomes. So do temporal-difference methods. The opposite is Monte Carlo, which uses complete returns and no bootstrapping. Bootstrapping makes updates faster and lower-variance and lets you learn before an episode ends, but it introduces bias and can interact badly with function approximation and off-policy training.