Designing an A/B test

Start from a decision, not a dashboard

Imagine you work on an online learning site. A product manager has an idea: add a “Continue learning” card to the home page so returning students jump straight back into their lesson. Everyone likes it. The tempting next move is to build it, ship it to everyone, and then watch the dashboard. A week later lessons-completed is up 3%, the team high-fives, and the card is declared a success.

We saw in the previous guide why this is a trap: when everyone gets the change, you have no honest comparison. Maybe lessons went up because it was exam season, or because of a marketing email, not because of the card. The fix is a randomized experiment: an A/B test that randomly splits users into a control group (who keep the current home page) and a treatment group (who see the new card), then compares the two. Random assignment makes the groups alike in every other way, so a difference between them can be credited to the card.

But here is the part beginners skip: an A/B test is only as good as its design, and the design happens before you write a single line of code. Designing the test means deciding, in advance, exactly what you will measure, how you will split users, what would count as success, and what would make you walk away. This guide is that pre-flight checklist. Do it well and the analysis later is almost boring; skip it and no amount of clever statistics can save you.

The single most clarifying question to start from is not “what should we measure?” but “what decision will this test let us make?” An experiment is a machine for answering a question, and a vague question yields a vague answer. If you cannot finish the sentence “If the result is ___, we will ___,” you are not ready to launch.

Choosing the overall evaluation criterion

The heart of the design is the metric you will judge the test by. We call it the overall evaluation criterion, or OEC — also just called the primary metric. It is the one number (occasionally a small, fixed set) that captures whether the change is genuinely good for users and for the business. The whole point of naming it now is to commit before you see the data, so you cannot later cherry-pick whichever metric happens to look good.

Why insist on essentially one metric? Because if you track twenty metrics and let yourself declare victory whenever any of them turns positive, you will almost always “win” by pure luck. With twenty independent metrics measured at the usual 5% threshold, the chance that at least one looks significant even when the change does nothing is about 64%. That is the multiple-comparisons trap, and the cleanest defense is to decide up front which single metric is the verdict.

A good OEC has four properties. It is sensitive (it actually moves when the change matters, within the time you can run the test). It is aligned with the long-term goal you really care about. It is hard to game. And it is measurable cleanly inside the experiment window. These often conflict. For our learning site, “clicks on the card” is sensitive and easy to measure but shallow and easy to inflate — a flashy card gets clicks even if nobody learns more. “4-week retention” is beautifully aligned with the real goal but far too slow to measure in a two-week test. A sensible compromise is a metric in between: lessons completed per weekly active user. It sits deep enough in the funnel to reflect real value, yet moves fast enough to read.

There is a famous warning here, called Goodhart's law: when a measure becomes a target, it stops being a good measure. The moment a team is told to maximize clicks, someone makes the card bigger, redder, and harder to dismiss — clicks soar, learning does not, and users are annoyed. This is exactly why you pick an OEC that is hard to game and pair it with the guardrails in the next section. The OEC tells you whether you won; the guardrails make sure you did not win by cheating.

Sometimes no single raw metric captures “good,” and teams build a composite OEC by combining a few normalized sub-metrics with weights. Introduced in words: take each sub-metric, rescale it to comparable units, multiply by a weight that reflects how much it matters, and add them up. In symbols, where each z is a standardized sub-metric and each w is its weight:

A composite OEC as a weighted sum of standardized sub-metrics. Be honest: the weights are human judgment calls, not facts from the data — choose them before the test and write down why.

1. List candidate metrics from shallow (clicks) to deep (long-term retention/revenue).
2. For each, ask: is it sensitive in our window? aligned with the real goal? gameable?
3. Pick the deepest metric that still moves reliably within the planned run length.
4. Pin exactly one OEC (or a tiny fixed composite) as the decision metric — and stop there.

Guardrail metrics: do no harm

Optimizing one number in isolation is dangerous, because a change can lift your OEC while quietly wrecking something else. A guardrail metric is a number you are not trying to improve but refuse to harm — like guardrails on a mountain road, they are not the destination, they just keep you from driving off a cliff. You decide them in advance, alongside the OEC.

Guardrails come in two flavors. Quality and trust guardrails protect the user experience: page load time, crash or error rate, and accessibility. Business guardrails protect the company: revenue per user, subscription cancellations, support-ticket volume. For the “Continue learning” card, two obvious guardrails are page load time (a heavier home page could slow everyone down) and the weekly cancellation rate (a pushy card might irritate people into leaving). If the card raises lessons completed but also slows the page by 400 milliseconds and nudges cancellations up, that is not a win — it is a trade you must make consciously, not by accident.

So the real decision rule has two parts, not one: ship only if the OEC moved up by a meaningful amount AND every guardrail stayed within its tolerance. Writing both halves down now prevents the all-too-human move of explaining away a guardrail regression after the fact (“oh, that latency hit is probably noise”).

The randomization unit: user, session, or cluster?

The randomization unit is the thing you flip the coin for — the entity you randomly assign to control or treatment. The usual choices are: the individual user, a single session or visit, a single request, or a whole cluster of people (a class, a school, a city). This sounds like a technicality, but it quietly determines both how trustworthy and how sensitive your test is.

The default for product experiments is the user. The reason is consistency: a single person should see the same experience every time they come back. If you randomized per session, a student might see the “Continue learning” card on Monday, lose it on Tuesday, and get it back on Wednesday. That flicker is confusing on its own, and it also smears the treatment and control together inside one person, making the comparison muddy. Assign by stable user id and the experience is coherent.

There is a genuine trade-off, though. Finer units (sessions, requests) give you many more units, which means more statistical power and a faster read — but at the cost of a flickering experience and contamination. Coarser units (clusters) are sometimes mandatory. When one user's treatment can affect another user's outcome — a social feature, a marketplace where buyers and sellers interact, or classmates who study together — randomizing individuals leaks the treatment across the boundary. We call this interference (or a spillover / network effect): the no-spillover assumption that lets us compare groups quietly breaks. The fix is to randomize whole clusters (e.g., assign entire schools to treatment or control) so the spillover stays inside one bucket.

Finally, the assignment itself should be deterministic and reproducible, not a fresh coin flip each request. The standard trick is to hash a stable id together with the experiment name, then read off a bucket. The same user always lands in the same group, anyone can recompute the assignment later, and adding a new experiment does not disturb the old ones.

# Deterministic, reproducible assignment (pseudocode)
bucket  = hash(user_id + ':' + experiment_id) % 100
variant = 'treatment' if bucket < 50 else 'control'
# Same user_id -> same bucket -> same variant, every time.

Write the hypothesis and success criteria first

Before the test goes live, write down — and ideally lock in a shared document — the full plan: the hypothesis, the OEC and guardrails, the size of effect you care about, how long you will run, how you will analyze, and the decision rule. This habit is called pre-registration, and it is the cheapest insurance in all of experimentation.

Why it matters is psychological as much as statistical. If you decide what counts as success only after seeing the data, every test succeeds — you will always find some metric, some sub-group, some week where things look good. Two named sins live here. p-hacking is trying analysis after analysis until something crosses the line. HARKing (Hypothesizing After the Results are Known) is inventing the hypothesis to match whatever you found, then presenting it as if you predicted it all along. Pre-registration disarms both because the hypothesis and the metric were fixed before any data existed.

State the hypothesis in plain words, and connect it to the formal machinery you met earlier. The null hypothesis is the boring default: the card makes no difference to the OEC. The alternative hypothesis is your bet: the card raises lessons-per-active-user. The test then asks a precise question — if the null were true and the card truly did nothing, how surprising would data this extreme be? That surprise is summarized by the p-value, and you compare it to a pre-chosen significance level (almost always 5%). Crucially, the p-value is NOT the probability the card works; it is the probability of data this extreme assuming the card does nothing. Fixing the significance level now, before the data, is part of pre-registering honestly.

Pre-registration also forces you to commit to the minimum detectable effect (MDE) and the sample size up front. The MDE is the smallest improvement that would actually be worth shipping — not the smallest you could detect, but the smallest you care about. This is also where effect size meets practicality: a 0.1% lift might be real and still not worth the engineering cost. The smaller the MDE you insist on catching, the more users you need, because tiny signals are buried in noise. As a rough rule of thumb for comparing two group averages — at the usual 5% significance and 80% power — the number of users you need per group is about sixteen times the variance divided by the squared effect:

A back-of-envelope sample size per group: n grows with the noise (σ², the variance) and shrinks fast with the effect you want to catch (Δ). Halving the MDE roughly quadruples the sample. The next guide derives this properly.

1. Write the hypothesis in one plain sentence (what changes, which metric, in which direction).
2. Fix the OEC, the guardrails, and each guardrail's tolerance.
3. Set the MDE (smallest effect worth shipping), the significance level, and the target power.
4. Compute the required sample size and the run length; commit to stopping then, not earlier.
5. Write the decision rule and the analysis method, and have a second person review it before launch.

Instrumentation and logging gotchas

An experiment can only measure what you logged. Data you forgot to capture is gone forever, so instrumentation — the events and fields your code records — is part of the design, not an afterthought. List every event the OEC and guardrails need (exposure, click, lesson-start, lesson-complete, page-load-time), and confirm each one actually fires correctly before you trust a single result.

The cheapest way to catch a broken pipeline is an A/A test: split users into two groups that get the exact same experience, and check that the analysis finds no difference. If an A/A test reports a “significant” effect, your randomization, logging, or analysis is broken — fix that before you ever run a real A/B. Run an A/A first, or run one continuously in the background as a smoke detector.

Three gotchas bite again and again. First, exposure logging, also called triggering: only count users who were actually in a position to see the change. If the “Continue learning” card only appears for returning students, including brand-new visitors who could never see it just dilutes the effect with people the test cannot move. Second, sample-ratio mismatch (SRM): you intended a 50/50 split but observe, say, 53/47. That gap means something is dropping users unevenly — a redirect that fails more often in one arm, bot traffic, or lost logs — and a test with SRM should not be trusted, full stop. Third, the usual data hygiene: duplicate events, bot filtering, timezone boundaries, and logging that silently drops rows under load.

-- Per-variant sanity check: is the split ~50/50, and what is the metric?
SELECT
  variant,
  COUNT(DISTINCT user_id)        AS users,        -- compare for SRM
  AVG(lessons_completed)         AS lessons_per_user
FROM experiment_exposures e
JOIN user_weekly_activity a USING (user_id)
WHERE experiment_id = 'continue_learning_v1'
GROUP BY variant;

A design template you can reuse

Everything above fits on a single page — an experiment one-pager you fill in before launch and keep forever. Reusing the same template turns design from a heroic effort into a habit, makes reviews fast, and (because it is versioned) makes every result reproducible months later when someone asks “wait, what exactly did we test?”

experiment: continue_learning_card_v1
owner: jovana
hypothesis: >
  Showing a 'Continue learning' card on the home page
  raises lessons completed per weekly active user.
oec: lessons_completed_per_wau          # the one decision metric
guardrails:
  - page_load_time_p95                  # must not increase materially
  - weekly_cancellation_rate            # must not increase materially
randomization_unit: user                # analyze per user too
variants:
  control:   current_home
  treatment: home_with_continue_card
mde: 0.02                  # +2% relative; smallest effect worth shipping
significance_level: 0.05
power: 0.80
sample_size_per_arm: 42000
planned_duration_days: 14  # do not stop early (no peeking)
analysis: two-group test on TRIGGERED users; report effect with 95% CI
decision_rule: >
  Ship only if the OEC interval is positive and excludes 0,
  AND no guardrail interval shows meaningful harm.

Notice what the template quietly protects you from, all decided before launch. Pre-committing to the run length defends against peeking (stopping the moment a metric looks good, which badly inflates false positives — the subject of a later guide). Naming a single OEC defends against metric soup. Specifying triggered users and the randomization unit defends against dilution and broken standard errors. And the decision rule, written in advance, defends against your future self's wishful thinking.

1. Decision written as one sentence (what result leads to what action).
2. Exactly one OEC chosen; guardrails and their tolerances listed.
3. Randomization unit chosen, and analysis uses the same unit.
4. MDE, significance level, power, sample size, and run length fixed.
5. Instrumentation verified with an A/A test; SRM check automated.
6. The whole one-pager reviewed and committed to version control.

That is the craft of designing an A/B test: most of the trustworthiness is bought before the experiment runs, by choosing the right metric, the right unit, sensible guardrails, and a decision rule you wrote while you were still honest. The next guide turns the MDE and sample size you committed to here into real numbers — the math of sample size and power — so you know exactly how long to run before you ever press start. Keeping these one-pagers together also gives you reproducibility and a growing institutional memory of what your team has already learned.