A/B testing
/ AY-bee TEST-ing /
Offline metrics tell you how a model scores on past data, but they can't tell you whether real people will actually respond better. A/B testing finds out by running a live experiment. You split your users at random into two groups: group A keeps the current version (the control), group B gets the new one (the variant). Then you watch what actually happens — clicks, purchases, time spent — and compare.
The magic ingredient is randomness. Because users are assigned to A or B by coin flip, the two groups are, on average, alike in every other way — age, mood, device, time of day. So if group B buys more, the new version is the most plausible cause, not some hidden difference between the groups. This is the same logic as a randomized clinical trial, which is why A/B testing is the closest thing the tech world has to a true cause-and-effect experiment.
Done right it's the gold standard for "does this change actually help?" But the pitfalls are everywhere and worth respecting. Peeking at results early and stopping the moment they look good inflates false positives. Tiny samples mislead. Running dozens of tests guarantees some will look significant by chance. And A/B tests measure short-term behavior on whoever's online now — they can miss long-term effects, novelty wearing off, or harm to users who weren't in the experiment. A clean A/B test answers a real question precisely; it does not answer every question.
A site tries a new "Buy now" button color. Half of visitors (random) see blue, half see green. After 100,000 visitors, green converts at 4.2% vs blue's 4.0%, and a significance test says the gap is unlikely to be luck. Green wins — but only for this button, this audience, this period.
Randomized split + significance test = a genuine cause-and-effect answer, scoped narrowly.
Decide your sample size and stopping point before you start, and resist "peeking." Repeatedly checking and stopping as soon as the result looks good dramatically inflates false positives — one of the most common ways well-intentioned A/B tests produce confident but wrong conclusions.