Experiment Design & A/B Testing
Overall Evaluation Criterion (OEC)
The single, agreed-upon metric (or carefully weighted combination of a few) that you decide BEFORE the experiment will determine whether the change wins or loses. It is the experiment's verdict in one number.
Picking the OEC up front protects you from the temptation to go fishing through dozens of metrics afterward until one looks good. A good OEC captures long-term value, not just a short-term click — for example revenue-per-user or weekly active days rather than “clicks on the new banner”, which a flashy-but-useless change can inflate.
A search engine uses “sessions per user per week” as its OEC, so a change that boosts clicks but makes people search more out of frustration would actually look bad.
Also called