ensemble learning

Ensemble learning is the machine-learning version of "two heads are better than one" — combine many imperfect models into one that's better than any of them alone. The classic intuition is the carnival jar of jellybeans: any single guess at how many it holds is wildly off, but average everyone's guesses and you land remarkably close to the truth. The errors point in random directions and cancel out, while a shared kernel of correct judgment survives. Ensembles engineer this same magic on purpose.

There are three main ways to combine models. Bagging trains many copies of the same kind of model on different random samples of the data and averages their votes — this is what a random forest does, and it mainly cuts down the wobble (variance) of jumpy models. Boosting builds models in sequence, each one focused on fixing the previous one's mistakes — this is what gradient boosting does, and it mainly cuts systematic error (bias). Stacking trains a final "manager" model to learn how best to blend the predictions of several different model types. Each strategy makes diverse mistakes and pools the survivors.

Ensembles are why the most accurate solutions on tabular data and in competitions are almost never a single model — they consistently squeeze out extra accuracy. But there's no free lunch. They cost more to train and run, they turn an interpretable model into a black box, and the trick only works if the members make different mistakes; combining ten copies of the same flawed model just repeats the same flaw ten times. Diversity, not quantity, is the active ingredient.