A/B testing when you do not have the traffic

The experimentation literature assumes you have millions of users. Most teams have thousands. Everything written for the first group is actively misleading for the second, because the standard advice — pick a metric, randomise, wait for significance — produces a coin flip when the sample size is not there.

Run the numbers first. At an 8% baseline, detecting a 5% relative lift at 80% power needs about 57,000 users per arm. At 2,000 users a week, that is a fifty-seven week experiment. The test is not underpowered; it is impossible.

So here is what is actually available.

1. Test bigger changes#

The most underrated option. Sample size scales with the inverse square of the effect: halving the effect you want to detect roughly quadruples the traffic you need.

A team with limited traffic should not be testing button colours. It should be testing the redesign, the new pricing page, the changed onboarding flow. Those move metrics by 20–40% when they work, and a 20% relative lift at an 8% baseline needs about 3,600 per arm — reachable in a few weeks.

This is a strategic point, not a statistical one: limited traffic should push you toward bolder changes, not more careful ones.

2. Move up the funnel#

Your conversion rate is 8% and needs enormous samples. But 60% of users reach the pricing page, and that is a metric with far more events and much less variance.

If your change plausibly affects an upstream step, measure the upstream step. It is a weaker claim — upstream movement does not guarantee downstream movement — but a well-measured weak claim beats an unmeasurable strong one, as long as you say which you have.

3. Use a continuous metric instead of a binary one#

Binary metrics throw away information. "Converted or not" collapses a user who spent $400 and one who spent $12 into the same value.

Revenue per user, or time spent, or items viewed, has more variance per observation but far more information, and often needs a smaller sample for the same power. Mind the heavy tail: use a bootstrap rather than a t-test, or analyse a winsorised version with the cap declared in advance.

4. Reduce variance with pre-experiment data#

CUPED (controlled experiment using pre-experiment data) is the single highest-leverage technique available to a small-traffic team. If you have each user's metric from before the experiment, you can subtract the predictable part of their behaviour and analyse the residual.

import numpy as np

def cuped_adjust(y, y_pre):
    """Returns a variance-reduced version of y using pre-period covariate y_pre."""
    theta = np.cov(y, y_pre)[0, 1] / np.var(y_pre)
    return y - theta * (y_pre - np.mean(y_pre))

Variance reductions of 30–50% are common for metrics with strong pre-period correlation, which translates to needing 30–50% less traffic. It costs one join and ten lines of code, and it is free power.

Stratifying randomisation on a pre-period variable achieves something similar at assignment time.

5. Switchback and interrupted time series#

When you cannot split users, split time. Alternate the treatment on and off in blocks — hours or days — and compare periods. This is standard in marketplaces where user-level randomisation contaminates the control anyway.

The costs: you need enough blocks (a dozen minimum), you must handle carryover between adjacent blocks, and any trend or seasonality confounds you unless the blocks are balanced across it. Randomise the block order rather than alternating strictly.

6. Sequential testing, declared in advance#

Fixed-horizon tests forbid peeking. Sequential methods — mSPRT, always-valid p-values, group sequential designs — permit continuous monitoring with valid error control, at the cost of some power.

For a small team this is genuinely useful: it lets you stop early when an effect is large, freeing traffic for the next test. The rule is that the method must be chosen before launch. Applying a sequential correction after peeking on a fixed-horizon test is not a correction.

7. Accept a Bayesian answer#

With small samples, "probability the variant is better, given a reasonable prior" is often more decision-useful than a p-value. It degrades gracefully: at n = 200 you get a wide posterior that honestly says "we do not know much", rather than a non-significant result that gets misread as "no effect".

The catch is the prior, and the discipline is to write it down before launch and be able to defend it.

8. Decide without an experiment, and say so#

Sometimes the honest answer is: this cannot be measured at our scale, so we will decide on judgement, ship it, and watch the guardrails.

That is a completely legitimate engineering decision. What is not legitimate is running an underpowered test, getting a null result, and reporting "no significant difference" as though it meant "no effect". It means "our instrument was not sensitive enough", and those two sentences lead to opposite decisions.

The order I would try them#

1. Test a bigger change.
2. Apply CUPED.
3. Move to a continuous metric or an upstream one.
4. Consider a switchback if the unit allows.
5. If none of that gets you there, ship on judgement and monitor.

Work it out on your own numbers with the Experiment Calculator before you build anything. Ten minutes there regularly saves a quarter.