Counting your tests

You meet it when#

A result comes back flat and someone slices it. Device, country, tenure band, plan, channel, new versus returning. That is not diligence — it is running more tests, and the arithmetic does not care about your intent.

Test 20 hypotheses at α = 0.05 with nothing real happening and you expect one "significant" result. The probability of at least one is about 64%.

The pattern#

python

def holm_bonferroni(p_values, alpha=0.05):
    """Rejection decision per test, in the original order.
    Strictly more powerful than Bonferroni with the same guarantee."""
    indexed = sorted(enumerate(p_values), key=lambda pair: pair[1])
    m = len(p_values)
    rejected = [False] * m
    for rank, (original_index, p) in enumerate(indexed):
        if p <= alpha / (m - rank):
            rejected[original_index] = True
        else:
            break          # once one fails, every larger p-value fails too
    return rejected

Holm dominates plain Bonferroni — same family-wise error guarantee, uniformly more power. There is no situation where you should prefer alpha/m for every test.

Choosing the correction#

Situation	Method
A handful of pre-declared tests	Holm–Bonferroni
Screening many hypotheses, expecting some real	Benjamini–Hochberg (controls false discovery rate)
One primary metric declared before launch	No correction needed — that is the point

What no correction can fix#

Corrections only cover the tests you report. If you tried nine metrics and reported the one that worked, applying a correction to that single test is not honest — it is arithmetic performed on a pre-selected result.

The defence is upstream and costs one line: write down the primary metric before you look. Everything else is exploratory and gets labelled as such, every time, in the write-up template itself so it is the default rather than something to remember.

The minimum, if you do nothing else#

State the count. "We looked at fourteen segments; this is the one that reached significance." Even with no correction applied, that sentence lets a reader discount appropriately. Omitting it is the part that is actually misleading.

Try it on clinical-trial — a real 4.1-point effect and eleven baseline covariates, which is exactly enough to find a spurious responder subgroup at p < 0.05.