Field note · June 4, 2023
Course note — Correlation, confounding, and Simpson's paradox
The gap between "these move together" and "this causes that" is where most bad decisions live. Here is how to see the gap, and what to do when you cannot close it.
Revised Correlation, confounding, and Simpson's paradox this week. The change was small and the reason was not.
The gap between "these move together" and "this causes that" is where most bad decisions live. Here is how to see the gap, and what to do when you cannot close it.
The first draft explained the mechanism and stopped. What was missing was the failure: the thing that happens when you get it wrong, described concretely enough that someone recognises it later. A lesson that only teaches the correct version leaves you unable to spot the incorrect one, which is the situation you will actually be in.
It sits in the Statistics you will actually use module, and the exercise runs against world-indicators. That pairing is deliberate: the dataset was built with the trap the lesson describes already in it, so the exercise fails in the instructive way rather than the confusing one.
Ordering matters here more than in most courses. It follows The five tests that cover most questions and leads into Designing an experiment that can succeed, and reading it out of sequence mostly works but costs you the setup.
Free means free, and it also means we can rewrite it whenever it is wrong. No edition, no errata PDF, nothing to repurchase. The whole course is 36 lessons and the fixes land the day we find them.