My experiment came back flat. Now what?

A question that came in more than once, so the answer goes here.

My experiment came back flat. Now what?

First, work out whether it was ever capable of detecting the effect you cared about. Compute the minimum detectable effect for the sample size you actually got. If your MDE was 5% and you were hoping to detect 1%, the result is "we learned nothing", not "no effect".

That is the most common outcome in practice and it is rarely reported that way.

Second, check for a sample ratio mismatch before interpreting anything. If the split is meaningfully off from what you configured, something in the assignment is broken and the comparison is not valid regardless of what it says.

Third, if the test was adequately powered and came back flat, believe it. A well-powered null is information — often more useful than a marginal win, because it closes a question rather than opening a follow-up.

What not to do, in order of how often it happens: slice by segment until something is significant, extend the test until it crosses the line, or quietly reframe the primary metric as a secondary one. All three are the same move, and all three produce a result that will not replicate. If you genuinely need to look at segments, decide which ones before the test starts and correct for the number of them.

The experiment calculator does the MDE arithmetic.