Experiment Design & A/B Testing
Sample Ratio Mismatch (SRM)
When the actual split of users between groups differs from what you intended — you planned a 50/50 test but ended up with, say, 52/48. A statistical check flags whether that gap is bigger than random chance can explain.
SRM is a loud alarm bell, not a metric to optimize: if the groups aren't even the size you randomly assigned, something is broken (a redirect dropping users, a logging bug, a bot filter hitting one side harder), and you should distrust the entire result until you find the cause. Always run an SRM check before reading any A/B test's outcome.
A test meant to be 50/50 across 1,000,000 users shows 510,000 vs 490,000 — a chi-square test gives a tiny p-value, so the experiment is invalid until the bug is found.
Also called