Uncertainty, sampling, and how much to trust a number

You compute a conversion rate of 4.2% from last week's data. Next week, with nothing changed, you would get a different number. Understanding how different — and why — is the difference between reporting a result and reporting noise.

Sampling variation, in one idea#

If you draw a sample from a population and compute a statistic, that statistic has its own distribution across all the samples you could have drawn. That is the sampling distribution, and its spread is the standard error.

For a mean, the standard error is:

$$ SE = \frac{s}{\sqrt{n}} $$

The √n is the whole economics of measurement: to halve your uncertainty you need four times the data. To get one more decimal place you need a hundred times. This is why "just collect more data" stops being a good answer surprisingly quickly.

The central limit theorem, and its actual limits#

The CLT says the sampling distribution of the mean approaches normal as n grows, whatever the population's shape. This is why so much of statistics works on data that is not itself normal.

The part that gets dropped in the retelling: how fast it converges depends on the population's skew. For a symmetric population, n = 20 is plenty. For a heavily right-tailed one — which describes most revenue data — n = 30 is nowhere near enough, and n = 1,000 may still leave a visibly skewed sampling distribution.

Open the Distribution Lab, pick lognormal, and set n = 5, then n = 200. The difference between those two pictures is the reason "n > 30 so the CLT applies" is folklore rather than a rule.

Confidence intervals, said carefully#

A 95% confidence interval is a procedure with a coverage property: if you repeated the whole study many times, 95% of the intervals you construct would contain the true value.

It is not "a 95% chance the true value is in this interval". The true value is fixed; the interval is what varies. This distinction sounds pedantic and matters in exactly one practical way: it stops you saying things like "there is a 95% chance the lift is between 1% and 6%", which invites a decision-maker to reason about it as a probability distribution over the truth — which it is not.

What an interval is genuinely for: communicating how much you know. A lift of "3% (95% CI: 2.4% to 3.6%)" and a lift of "3% (95% CI: −4% to 10%)" are the same point estimate and completely different findings. Lead with the interval.

The bootstrap: one tool for almost everything#

When you cannot remember the formula for a statistic's standard error — or there is not one, as for a median or a 90th percentile — resample.

import numpy as np

def bootstrap_ci(data, statistic=np.median, n_boot=10_000, alpha=0.05, seed=0):
    """Percentile bootstrap interval. Works for any statistic you can compute."""
    rng = np.random.default_rng(seed)
    data = np.asarray(data)
    n = len(data)
    estimates = np.empty(n_boot)
    for i in range(n_boot):
        sample = data[rng.integers(0, n, n)]     # resample WITH replacement
        estimates[i] = statistic(sample)
    return np.quantile(estimates, [alpha / 2, 1 - alpha / 2])

bootstrap_ci(fares, np.median)
bootstrap_ci(fares, lambda x: np.quantile(x, 0.9))
bootstrap_ci(revenue, np.mean)          # honest about heavy tails

The bootstrap's assumption is that your sample resembles the population. It cannot rescue a biased sample — resampling a broken sample a million times gives you a very precise estimate of the wrong number.

Bias beats variance, and it does not shrink with n#

Sampling error is random and shrinks as √n. Bias is systematic and does not shrink at all. A survey that only reaches people who answer surveys is exactly as biased at n = 1,000,000 as at n = 1,000, and it will have a beautifully tight confidence interval around the wrong answer.

The biases that actually occur in this work:

Survivorship. Analysing accounts that are still active tells you about survivors, not about accounts. Any "our customers love X" analysis over current customers has this.

Selection. People choose whether to appear in your data. Ratings, reviews, and support tickets are all self-selected. movie-ratings is built to demonstrate this: people mostly rate films they chose to watch, so average rating measures popularity as much as quality.

Informative missingness. The reason a value is missing relates to the value. data-job-postings hides salary more often for junior and non-US roles, so an average over disclosed salaries runs high.

Left truncation. Your data starts when logging started, not when the phenomenon did. Any "cohorts have got worse over time" finding should be checked against a tracking change.

Sample size, practically#

For a proportion, the standard error is at its worst when p = 0.5, giving a conservative rule for the margin of error at 95%:

$$ n \approx \frac{1}{MOE^2} $$

A ±3% margin needs about 1,100 observations. A ±1% margin needs about 10,000. A ±0.1% margin needs a million — which is why "we want to detect a 0.1% lift" is usually a request to abandon the experiment rather than to run it.

Note that this depends on the absolute count, not the fraction of the population. Sampling 1,100 people from a country of 60 million gives the same precision as sampling 1,100 from a town of 30,000. The population size is nearly irrelevant, which almost nobody believes on first hearing.