model ensembling for classification

Ensembling combines the predictions of several models into one, almost always improving classification accuracy and calibration over any single member. The standard recipe for classifiers is soft voting: average the predicted probability vectors of the members and take the argmax of the average. Hard voting, where each model casts one vote for its top class and the majority wins, is an alternative but discards the confidence information that soft voting exploits.

It works because independent models make partly independent errors. If each model is individually accurate but their mistakes are not perfectly correlated, averaging cancels the uncorrelated error component while preserving the shared correct signal, which is the same bias-variance logic that makes the average of noisy estimates more reliable than any one. The benefit therefore grows with the diversity of the members, which you encourage by varying architectures, random seeds, data orderings, augmentations, or training subsets.

The headline cost is linear: an ensemble of k models needs k times the storage and inference compute, which is often unacceptable in production. This motivates cheaper approximations such as snapshot ensembles, which save several checkpoints from one training run, and knowledge distillation, which compresses an ensemble's averaged predictions into a single fast student network that retains much of the gain.