Most of our experiments could never have worked

I went back through two years of experiments at a previous company and computed, for each, the effect it had power to detect at the sample size it actually reached.

Of 61 tests, 34 could not have detected anything smaller than a 20% relative lift at 80% power. The hypotheses they were testing were, in the write-ups, mostly in the 2–5% range.

Those 34 tests could not succeed. They were coin flips with a scientific veneer, and nearly all of them concluded "no significant difference" — which was then read as "the feature did nothing" rather than "our instrument was not sensitive enough". Several perfectly reasonable ideas were killed on that basis.

The arithmetic that would have prevented it takes ten minutes:

sample_size_per_arm(baseline=0.086, mde_relative=0.05)   # ~57,000 per arm
sample_size_per_arm(baseline=0.086, mde_relative=0.20)   # ~3,600 per arm

Halving the effect you want to detect roughly quadruples the traffic you need. That single fact reframes the whole practice for a team without millions of users: you should be testing bigger changes, not more careful ones.

What we do now, and it is one line in the experiment doc: state the MDE and the required sample size before launch. If the required sample exceeds what the traffic can deliver in a reasonable window, that is the finding. Do not run the test. Either raise the effect you are willing to detect, or ship on judgement and say so.

Shipping on judgement and saying so is a completely respectable engineering decision. Running an underpowered test and reporting the null is not.

The calculator does this arithmetic. It takes about a minute.