My experiment came back flat. Now what?
First, work out whether it was ever capable of detecting the effect you cared about. Compute the minimum detectable effect for the sample size you act
Tag
experiments
13 pieces tagged “experiments”, newest first.
Counting your tests, in short
Twenty segments at α = 0.05 gives you a one-in-two chance of a false finding. Correct for it, or at minimum say how many you ran.
Pattern: Counting your tests
Twenty segments at α = 0.05 gives you a one-in-two chance of a false finding. Correct for it, or at minimum say how many you ran.
A proportion with an interval, in short
A percentage without a denominator is not a finding. The Wilson interval behaves sensibly at small n, where the normal approximation falls apart.
Pattern: A proportion with an interval
A percentage without a denominator is not a finding. The Wilson interval behaves sensibly at small n, where the normal approximation falls apart.
Counting your tests, in short
Twenty segments at α = 0.05 gives you a one-in-two chance of a false finding. Correct for it, or at minimum say how many you ran.
CUPED gave us back 40% of our sample size for ten lines of code
Variance reduction using pre-experiment data, explained without the paper.
Most of our experiments could never have worked
We audited two years of tests. Over half were incapable of detecting the effect they were looking for.
Difference-in-differences, in short
When a change hits one group and not another, compare the change over time rather than the levels. Any time-invariant difference cancels out.
Pattern: Difference-in-differences
When a change hits one group and not another, compare the change over time rather than the levels. Any time-invariant difference cancels out.
Difference-in-differences, in short
When a change hits one group and not another, compare the change over time rather than the levels. Any time-invariant difference cancels out.
A proportion with an interval, in short
A percentage without a denominator is not a finding. The Wilson interval behaves sensibly at small n, where the normal approximation falls apart.
We p-hacked ourselves and caught it in review
Fourteen segments, one significant result, and the correction that made it go away.