Experiment Design & A/B Testing
CUPED (Variance Reduction)
A technique that makes A/B tests more sensitive by subtracting out predictable, pre-experiment noise. The name stands for Controlled-experiment Using Pre-Experiment Data: you use each user's behaviour from before the test (say, last week's spend) to adjust their during-test metric, removing differences that have nothing to do with your change.
Because the test statistic depends on variance, shrinking the noise is like turning up the resolution — you can detect the same effect with substantially fewer users or in less time, often cutting required sample by a third or more. It works best when pre-period behaviour strongly predicts the metric, and crucially it does not bias the result, only sharpens it.
Also called