gradient boosting

Gradient boosting builds a strong predictor out of a long line of weak ones, each one apologizing for the last one's mistakes. Picture a student tackling a problem set: the first attempt gets a lot wrong, so they study exactly the questions they missed; the second pass fixes some of those, leaving a smaller pile of errors to attack next; and so on. Each new model is trained specifically on what the running total still gets wrong, and the team grows smarter one focused correction at a time.

Concretely, you usually use small decision trees as the weak learners. The first tree makes a rough prediction. You then compute the residuals — how far off each prediction was — and train the next tree not on the original target but on those leftover errors. Add its (shrunken) correction to the running prediction, recompute the new, smaller errors, and train another tree on those. "Gradient" refers to using calculus to point each tree in the direction that most reduces the error; "boosting" is the act of stacking these weak learners into one strong one.

Boosting is one of the most accurate techniques for everyday tabular data and routinely wins data-science competitions. But it is more delicate than a random forest: because trees are added sequentially to chase errors, push it too far and it will eventually start fitting the noise — overfitting — so you must rein it in with a small learning rate, shallow trees, and early stopping. It also trains in sequence rather than in parallel, which can make it slower to fit. Random forests average independent trees to cut variance; boosting adds dependent trees to cut bias — same ingredient, opposite strategy.