Reader question — Reading a result without fooling yourself

Reader question on Reading a result without fooling yourself, and the answer belongs somewhere more findable than an email.

The experiment finished. Here is the order to look at things in, and the specific ways a real-looking result turns out not to be.

The question was, roughly, "when does this stop applying?" — which is the right question and the one lessons routinely fail to answer. Every technique has a range of validity, and stating it is what separates a lesson from a recipe.

It sits in the Experiments and causal claims module, and the exercise runs against ab-test-checkout. That pairing is deliberate: the dataset was built with the trap the lesson describes already in it, so the exercise fails in the instructive way rather than the confusing one.

Ordering matters here more than in most courses. It follows Designing an experiment that can succeed and leads into Causal claims when you cannot randomise, and reading it out of sequence mostly works but costs you the setup.

Free means free, and it also means we can rewrite it whenever it is wrong. No edition, no errata PDF, nothing to repurchase. The whole course is 36 lessons and the fixes land the day we find them.