Holdout Group

A slice of users deliberately kept on the old experience for a long stretch — weeks or months — even after a change has launched to everyone else, so you can measure its true long-term, accumulated impact.

Individual A/B tests each show a small short-term lift, but those gains can fade, overlap, or even cancel out over time. A long-running holdout answers the bigger question “across everything we shipped this quarter, how much better off are users than if we'd shipped nothing?” The cost is that the held-out users miss improvements, so holdouts are usually small and time-limited.