XGBoost
/ EKS-jee-boost /
XGBoost is not a new idea so much as a superbly engineered version of an old one. It is gradient boosting — the build-a-team-of-trees-that-fix-each-other's-errors method — implemented so carefully that it trains fast, uses memory frugally, and resists overfitting better than the textbook version. Think of it as the difference between a homemade go-kart and a tuned race car: same engine principle, vastly better in practice. Released around 2014, it became the tool that won competition after competition on tabular data.
Its improvements are mostly practical craftsmanship layered onto plain boosting. It bakes regularization (a penalty on overly complex trees) directly into what each tree tries to optimize, so the model is nudged toward simplicity from the start. It uses cleverer math to decide each split, handles missing values automatically by learning which way to send them, and is heavily optimized to run across many processor cores and large datasets without choking. The result keeps boosting's accuracy while taming its fiddliness and its slowness.
Where it shines is structured, tabular data — spreadsheets of rows and columns like transactions, sensor readings, or medical records — and there it is still, years on, one of the strongest off-the-shelf choices, alongside cousins like LightGBM and CatBoost. What it is not is a tool for raw images, audio, or free text; those belong to neural networks. And it inherits boosting's caveats: many knobs to tune, and a real risk of overfitting if you let it run unchecked. Powerful, but not a cure-all.
A bank scores 2 million credit-card transactions for fraud, each with 80 features and many blanks. XGBoost trains in minutes across all the computer's cores, learns on its own which way to route the missing values, and flags the riskiest 0.1% — the kind of fast, accurate tabular workhorse it's famous for.
Boosting's accuracy, engineered for speed, scale, and built-in regularization.
XGBoost is an implementation of gradient boosting, not a different algorithm — the conceptual leap is boosting itself. On rows-and-columns data it often beats deep learning, but on raw images, sound, or text the advantage flips to neural networks.