Reading a result without fooling yourself

The test has run. Before you compute the lift, work through this in order — because once you have seen the headline number, everything after it is motivated reasoning.

1. Sanity checks first, results second#

Sample ratio. Did the arms get the traffic you assigned? A significant deviation means assignment or logging is broken and nothing downstream is trustworthy. This is a stop-the-line condition, not a caveat.

Exposure sanity. Did each arm actually see what it was meant to? A surprising number of "no effect" results turn out to be a feature flag that never flipped for 30% of the treatment arm.

Pre-period balance. The arms should look identical before exposure.

Guardrails. Any unacceptable harm? If latency doubled, the conversion lift is irrelevant.

Only now look at the primary metric.

2. Effect size and interval, before the p-value#

Conversion: control 8.61%, variant B 10.09%
Absolute lift: +1.48pp
Relative lift: +17.2%
95% CI on the absolute difference: +0.55pp to +2.41pp
n: 1,993 / 2,010

Read the interval, not the point estimate. This one excludes zero and its lower bound (+0.55pp) is above the MDE you set — a clear result. Compare with an interval of −0.4pp to +3.4pp, which has the same point estimate and tells you almost nothing.

That last case matters. "We can rule out any effect larger than 0.3%" is valuable knowledge that stops a team relitigating the idea every quarter.

3. Then the ways it can still be wrong#

Novelty and primacy. Users react to change, not to the design. Novelty inflates a new thing's performance early; primacy depresses it while people relearn a familiar flow. Both fade within one to three weeks.

Diagnose by plotting the effect by exposure week. If the lift decays monotonically, you are looking at novelty. ab-test-checkout has this in variant A on purpose — strong in week one, gone by week three.

Peeking. If anyone looked daily and the test was stopped soon after it crossed significance, the reported p-value is not the real one. The honest fix is to say the stopping rule was not pre-declared.

Interference. Did the treatment affect the control? Shared inventory, shared support queue, shared recommendation model, users who talk to each other. If yes, the comparison is between contaminated groups and the effect is understated.

Segment shopping. Sliced by device and found significance on tablets only? With five segments and two metrics you have run ten tests. Correct, or label it exploratory.

Metric mismatch. Did conversion rise while revenue per user fell? A discount does that. Always look at the value metric, not only the rate metric.

Survivorship in the denominator. If the treatment changes who reaches the point of measurement, the two arms' denominators are different populations. This one is subtle and common in funnel experiments.

4. Heavy-tailed metrics need different maths#

Revenue per user is zero-inflated (most users spend nothing) and heavy-tailed (a few spend a great deal). A t-test on it will happily return a p-value that means very little.

Better options:

Bootstrap the difference in means. Makes no distributional assumption.

import numpy as np

def bootstrap_diff(a, b, n_boot=10_000, seed=0):
    rng = np.random.default_rng(seed)
    a, b = np.asarray(a), np.asarray(b)
    diffs = np.empty(n_boot)
    for i in range(n_boot):
        diffs[i] = (rng.choice(b, len(b), replace=True).mean()
                    - rng.choice(a, len(a), replace=True).mean())
    return diffs.mean(), np.quantile(diffs, [0.025, 0.975])

Decompose it. Analyse conversion rate and average order value among converters separately. Two clean questions instead of one messy one, and the decomposition usually tells you why the metric moved.

Winsorise, declared in advance. Cap at the 99th percentile to limit the influence of a handful of whales. Legitimate only if decided before seeing the data, and stated in the write-up.

5. Write it up so it survives#

A good experiment write-up is short and has this shape:

Decision: ship variant B.

Variant B raised checkout conversion from 8.61% to 10.09%, an absolute lift of 1.48pp (95% CI: 0.55 to 2.41pp) over 4,003 users across 28 days. Revenue per user moved consistently with conversion (+14.8%, CI +2.1% to +27.9%), so the lift is not discount-driven.

Sample ratio check passed (p = 0.71). No guardrail regressions; p95 latency unchanged. The effect is stable across all four exposure weeks, so this is not novelty.

Variant A was also tested and is not recommended: it led in week one and regressed to control by week three, consistent with a novelty effect.

Exploratory (not corrected, hypothesis-generating only): the effect appears larger on mobile than desktop.

Decision first. Interval next. Sanity checks stated. Exploratory findings explicitly labelled. That write-up is reviewable by someone who was not involved, which is the actual standard.