Everything you know about p-values, restated carefully

I have watched competent people — including me — say things about p-values that are not true, in meetings where a decision was being made. The concept is genuinely slippery. This is an attempt at the careful version, without the hedging that usually makes these explanations useless.

The definition#

A p-value is: the probability of observing data at least as extreme as what you observed, assuming the null hypothesis is true.

Every word is load-bearing. In particular, "assuming the null hypothesis is true" means the p-value is computed inside a world where there is no effect. It tells you how surprising your data would be in that world. It cannot tell you the probability that you are in that world, because it started by assuming you were.

Four things it is not#

Not the probability the null is true. This is the big one. P(data | no effect) is not P(no effect | data). Converting between them requires a prior, which the p-value does not have and does not use.

Not the probability your result is a fluke. A restatement of the same error, and the one that most often reaches a slide.

Not a measure of effect size. With 40 million users you can get p < 0.001 on a 0.02% lift. Significance is a statement about evidence against a null, not about magnitude.

Not a measure of importance. Statistical significance and practical significance are unrelated quantities that share a word.

What it is actually good for#

One thing: a rough, conventional filter against being fooled by noise. In a world with no effect, you will see p < 0.05 about 5% of the time. That is a useful discipline against pattern-matching on randomness, and it is essentially the whole benefit.

It is a filter, not a finding. Passing it means "worth taking seriously". It does not mean "true", and it certainly does not mean "large".

The 0.05 threshold is a convention, not a law#

Fisher suggested it as a convenient rule of thumb and later regretted how rigidly it was adopted. There is nothing special about 5%. In a context where a false positive is expensive — shipping a change that will be hard to reverse — 0.01 might be right. In an exploratory screen where a false negative costs more, 0.10 might be.

What matters is choosing before you look, and saying which you chose.

Multiple comparisons: how honest people produce false findings#

Test 20 things at α = 0.05 with nothing real going on and you expect one significant result. This is not fraud, it is arithmetic, and it happens constantly because slicing a result by segment feels like diligence rather than like running more tests.

The defences, in order of how often you should reach for them:

1. Declare the primary metric before launch. One metric. Everything else is exploratory and gets labelled as such.
2. Correct when you genuinely have a family of tests. Holm–Bonferroni if you want to control the chance of any false positive; Benjamini–Hochberg if you are screening many hypotheses and expect some to be real.
3. Report the number of tests you ran. Even without a correction, "we looked at 14 segments" lets the reader discount appropriately. Omitting it is the actual dishonesty.

What to report instead#

The order that keeps people honest:

1. The effect size, in units someone cares about. "Conversion rose 1.5 percentage points."
2. The confidence interval. "95% CI: 0.6 to 2.4 points."
3. The sample size.
4. The p-value, if at all.

An interval does everything a p-value does and more. If it excludes zero, you have significance. But it also tells you the magnitude and the precision, which is what the decision actually needs. A result of "+3% (CI: −4% to +10%)" and a result of "+3% (CI: 2.4% to 3.6%)" have the same point estimate and completely different meanings, and the p-value alone cannot distinguish them.

The confidence interval also needs care#

A 95% confidence interval is a claim about the procedure: repeat the study many times, and 95% of the intervals constructed this way contain the true value. It is not "a 95% chance the truth is in this interval" — the truth is fixed, the interval is what varies.

In practice this distinction rarely changes a decision, and I would rather someone used intervals slightly loosely than used p-values correctly. But it is worth knowing which claim you are making, because someone will eventually ask.

The practical summary#

• Set the primary metric and the threshold before you look.
• Report the interval; treat the p-value as a footnote.
• Count your tests and say how many you ran.
• Never say "we proved" or "there is no effect". Say "we can rule out effects larger than X", which is both true and more useful.

Try it on clinical-trial, which has a real 4.1-point effect and eleven baseline covariates — exactly enough to go subgroup hunting and find something spurious at p < 0.05. That exercise teaches this faster than any explanation.