Patterns / Measuring

Bootstrap any statistic

When you cannot remember the standard error formula — or there is not one — resample. It works for medians, percentiles and ratios alike.

Working ·Statistics

You meet it when#

You need an interval around something that is not a mean: a median, a 90th percentile, a ratio of two sums, a difference between two groups' medians. Closed-form standard errors for these are either awkward or nonexistent.

The pattern#

python
import numpy as np

def bootstrap_ci(data, statistic=np.median, n_boot=10_000, alpha=0.05, seed=0):
    """Percentile bootstrap. Works for any statistic you can compute."""
    rng = np.random.default_rng(seed)
    data = np.asarray(data)
    n = len(data)
    estimates = np.empty(n_boot)
    for i in range(n_boot):
        sample = data[rng.integers(0, n, n)]      # resample WITH replacement
        estimates[i] = statistic(sample)
    return np.quantile(estimates, [alpha / 2, 1 - alpha / 2])

bootstrap_ci(fares, np.median)
bootstrap_ci(fares, lambda x: np.quantile(x, 0.9))
bootstrap_ci(revenue, np.mean)     # honest where a t-test is not

Three things matter: sample with replacement, sample the same n as the original, and seed it so the picture is reproducible.

For a difference between groups#

python
def bootstrap_diff(a, b, n_boot=10_000, seed=0):
    rng = np.random.default_rng(seed)
    a, b = np.asarray(a), np.asarray(b)
    diffs = np.empty(n_boot)
    for i in range(n_boot):
        diffs[i] = (rng.choice(b, len(b), replace=True).mean()
                    - rng.choice(a, len(a), replace=True).mean())
    return diffs.mean(), np.quantile(diffs, [0.025, 0.975])

This is the right tool for revenue per user in an experiment: zero-inflated and heavy-tailed, so the sampling distribution of the mean is not normal at realistic sample sizes and a t-test's interval is too narrow.

What it cannot do#

The bootstrap assumes your sample resembles the population. It cannot rescue a biased sample — resampling a broken sample ten thousand times gives you a very precise estimate of the wrong number.

It also struggles at the extremes: an interval around a maximum, or a statistic dominated by a single observation, is not something resampling can stabilise.