The five tests that cover most questions

You do not need to memorise thirty tests. You need to recognise a handful of question shapes and know what each one's assumptions are actually protecting against.

What a p-value is, and what it is not#

Is: the probability of seeing data at least this extreme if the null hypothesis were true.

Is not: the probability the null hypothesis is true. Not the probability your result was a fluke. Not a measure of effect size. Not a measure of importance.

A p-value of 0.03 with a 0.02% lift on 40 million users is statistically significant and commercially irrelevant. A p-value of 0.11 with a 9% lift on 400 users might be the most interesting thing you learn this quarter. Effect size and interval first; p-value as a footnote.

The decision table#

That last row is not a joke. If you can compute the statistic, you can bootstrap an interval for it, and an interval answers the question better than a test does.

Two proportions#

The single most common test in product work.

from math import sqrt
from statistics import NormalDist

def two_proportion_test(x1, n1, x2, n2):
    """Returns (lift, 95% CI on the difference, two-sided p)."""
    p1, p2 = x1 / n1, x2 / n2
    pooled = (x1 + x2) / (n1 + n2)
    se_null = sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))
    z = (p2 - p1) / se_null
    p_value = 2 * (1 - NormalDist().cdf(abs(z)))

    # The interval uses the unpooled SE: it is about the difference, not the null.
    se_diff = sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
    return (p2 - p1), (p2 - p1 - 1.96 * se_diff, p2 - p1 + 1.96 * se_diff), p_value

The pooled/unpooled distinction trips people up. Pooled variance under the null for the test statistic; unpooled for the interval, because the interval is not conditioned on the null being true.

Welch's t-test, and why not Student's#

Comparing two means. Use Welch's version, which does not assume equal variances, essentially always. It is barely less powerful when variances are equal and substantially more correct when they are not — and you rarely know in advance. Most libraries default to Student's; override it.

from scipy import stats
stats.ttest_ind(group_a, group_b, equal_var=False)   # Welch

The t-test assumes the sampling distribution of the mean is roughly normal, not that the data is. For mildly skewed data at reasonable n, fine. For revenue per user — zero-inflated and heavy-tailed — not fine, and a bootstrap is the honest choice.

Chi-square#

For association between two categorical variables. Assumes expected counts of at least about 5 per cell; below that use Fisher's exact test. A chi-square on a 2×2 table is equivalent to the two-proportion z-test, so use whichever expresses your question more naturally.

Multiple comparisons: the failure mode nobody plans for#

Test 20 hypotheses at α = 0.05 with nothing true and you expect one "significant" result. Test 100 subgroups and you will find five.

This is the most common way an honest analyst produces a false finding, because it does not feel like cheating. It feels like exploring.

Corrections, in increasing sophistication:

• Bonferroni — use α/m. Simple, very conservative, fine for small m.
• Holm–Bonferroni — strictly more powerful than Bonferroni, same guarantee. Use this instead; there is no reason not to.
• Benjamini–Hochberg — controls the false discovery rate rather than the family-wise error rate. Correct choice when you are screening many hypotheses and expect some to be real.

def holm_bonferroni(p_values, alpha=0.05):
    """Returns a rejection decision per test, in the original order."""
    indexed = sorted(enumerate(p_values), key=lambda pair: pair[1])
    m = len(p_values)
    rejected = [False] * m
    for rank, (original_index, p) in enumerate(indexed):
        if p <= alpha / (m - rank):
            rejected[original_index] = True
        else:
            break               # once one fails, all larger p-values fail too
    return rejected

Practical significance, always#

Before any test, write down the smallest effect that would change a decision. If your test cannot detect that effect at your sample size, the test cannot succeed and should not be run. If it detects something much smaller than that, "significant" is the wrong word to use in the summary.

This one habit — deciding the minimum interesting effect first — prevents more bad conclusions than every correction procedure combined.