Sample Size Calculation
Working out, before you launch, how many users each group needs so the experiment has a fair chance of detecting the effect you care about. It balances four things: the effect size you want to catch (MDE), how noisy the metric is (its variance), and your tolerances for false alarms and missed effects.
Run a test too small and you will likely see “no significant difference” even when a real improvement exists — a wasted experiment. Run it absurdly large and you slow down decisions. The calculation turns these trade-offs into a concrete number, which you then translate into how long the test must run given your daily traffic.
A rough rule of thumb: required sample per group grows with variance and shrinks with the square of the effect size — halving the MDE roughly quadruples the users you need.