Designing an experiment that can succeed

The most common outcome of an online experiment is "no significant difference", and the most common reason is not that the feature did nothing. It is that the test never had the statistical power to see the effect it was looking for.

That is decidable in advance. Ten minutes with a calculator, before anyone writes code.

The four quantities#

They are linked: fix any three and the fourth follows.

Baseline rate. What the control does today. Measure it; do not estimate it.

Minimum detectable effect (MDE). The smallest change you would act on. Not the change you hope for — the smallest one that would change a decision.

Significance level (α). Your tolerance for a false positive. Conventionally 0.05.

Power (1 − β). Your probability of detecting a real effect of MDE size. Conventionally 0.80, which means a one-in-five chance of missing a real effect. That is a much worse guarantee than most people realise, and 0.90 is often worth the extra traffic.

from math import sqrt
from statistics import NormalDist

def sample_size_per_arm(baseline: float, mde_relative: float,
                        alpha: float = 0.05, power: float = 0.8) -> int:
    """Users needed PER ARM for a two-sided test of two proportions."""
    p1 = baseline
    p2 = baseline * (1 + mde_relative)
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_beta = NormalDist().inv_cdf(power)
    p_bar = (p1 + p2) / 2
    numerator = (z_alpha * sqrt(2 * p_bar * (1 - p_bar))
                 + z_beta * sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return int(numerator / (p2 - p1) ** 2) + 1

sample_size_per_arm(0.086, 0.05)    # 5% relative lift → ~57,000 per arm
sample_size_per_arm(0.086, 0.10)    # 10% relative lift → ~14,300 per arm
sample_size_per_arm(0.086, 0.20)    # 20% relative lift → ~3,600 per arm

Look at those numbers. Halving the effect you want to detect roughly quadruples the traffic you need. This is the single most useful fact in experiment design, and it explains why small teams should be testing big changes rather than button colours.

Randomisation unit#

Randomise on the unit that experiences the change and matches your metric's grain.

User is the default and usually right. A user gets one experience across sessions and devices, which prevents the jarring inconsistency of session-level assignment.

Session gives more units and therefore more power, but breaks whenever the change is memorable — a user seeing a different checkout on Tuesday than Monday.

Cluster (account, household, city) when interference exists between units. If your change affects a marketplace's supply, randomising buyers individually contaminates the control, because they compete for the same inventory. This is interference, and it silently biases every marketplace and social experiment run on individual randomisation.

Guardrails and the primary metric#

Choose one primary metric before launch. Write it down. One.

Then choose three to five guardrail metrics you are unwilling to damage: latency, error rate, revenue per user, unsubscribe rate. Guardrails are not judged by significance — they are judged by whether the interval rules out an unacceptable harm.

Everything else is exploratory, and results from it generate hypotheses rather than conclusions. Label them that way in the write-up, every time, or the exploratory finding with p = 0.04 becomes the headline.

Before you trust anything: the sanity checks#

Sample ratio mismatch (SRM). You assigned 50/50; you observed 50.4/49.6. On a million users that is not chance — it means assignment or logging is broken, and every downstream number is suspect.

from scipy import stats
observed = [500_412, 499_588]
stats.chisquare(observed)      # p < 0.01 here → stop and investigate

SRM is the highest-value check in experimentation because it catches a whole class of instrumentation bugs, and it is one line. Run it first, before looking at any result.

A/A test. Run the pipeline with both arms identical. You should get a significant result about 5% of the time. If you get one much more often, your variance estimate is wrong — usually because of the analysis-unit mismatch above.

Pre-period balance. Compare the arms on the weeks before exposure. They should be indistinguishable. If they are not, randomisation is broken.

The duration decision#

Run for whole weeks — always. Weekday and weekend populations differ, and a test that runs Tuesday to Sunday is comparing an unbalanced mix.

Run at least one full business cycle, usually two weeks. Novelty effects fade, and a change that wins in week one and loses in week three is a change that loses. ab-test-checkout has this built in: variant A looks strong in its first week and regresses to control afterwards.

And do not peek. Checking daily and stopping when it goes significant inflates your false positive rate to somewhere around 20–30%, not 5%. If you need to stop early, use a method designed for it — sequential testing or a Bayesian approach with a pre-declared decision rule — and declare it before launch, not after you see the data.