Correlation, confounding, and Simpson's paradox

Everyone can recite "correlation is not causation" and almost nobody acts on it, because in the moment the correlation is right there and the causal story is so easy to tell.

What correlation actually measures#

Pearson's r measures linear association only, on a scale from −1 to 1. Its failure modes are systematic, not exotic:

• A perfect parabola has r ≈ 0. Strong relationship, zero linear correlation.
• One extreme point can carry a correlation on its own, or destroy it.
• Two clusters with no within-cluster relationship produce a strong overall r that describes nothing.

Spearman's ρ correlates the ranks instead, capturing any monotone relationship. A large gap between Pearson and Spearman means the relationship is monotone but curved — which is a finding, and a hint that a transform is in order.

Confounding#

X and Y both correlate because Z drives both.

Ice-cream sales correlate with drownings. Temperature drives both. Nobody is fooled by that example, and everyone is fooled by its business equivalents:

• Users of feature X retain better → people who were already engaged found feature X.
• Customers contacted by sales convert more → sales contacted the promising ones.
• Support tickets correlate with churn → both are downstream of a bad experience.

Each has a plausible causal story, a real correlation, and a confounder large enough to explain the whole effect.

Simpson's paradox#

The most dramatic version: a relationship that reverses when you disaggregate.

world-indicators contains one deliberately. Pooled across all thirty countries, internet penetration correlates negatively with gains in life expectancy. Within every individual country, the correlation is positive. Both are correct arithmetic.

The mechanism: countries with already-high internet penetration are rich countries with life expectancy near its ceiling, so they improve slowly. Countries with low penetration start low and improve fast. The between-country differences swamp the within-country relationship, and pooling mixes the two.

# Pooled: negative.
df[["internet_pct", "life_gain"]].corr().iloc[0, 1]

# Within country: positive, for essentially every country.
df.groupby("country")[["internet_pct", "life_gain"]] \
  .corr().unstack().iloc[:, 1].describe()

Which is right? Neither, without a question. If you are asking what happens when a given country gets more connected, the within-country estimate is relevant. If you are describing how countries differ, the pooled one is. The paradox is not a trick — it is two different questions that look like one.

The generalisable defence: whenever you compute an aggregate relationship, compute it within your most important grouping too. If the two disagree, you have learned something important and were about to publish something wrong.

Colliders: the subtler trap#

Confounding is a common cause. A collider is a common effect, and conditioning on one manufactures correlation that is not there.

Suppose talent and luck both cause someone to be hired, and are independent in the population. Among hired people they will correlate negatively — a low-talent hire must have been lucky. Restrict your analysis to hires and you have invented a relationship.

This happens constantly in business data because almost every dataset is already conditioned on something: only customers who purchased, only users who logged in, only accounts that survived. Sample selection is conditioning, and conditioning on a collider creates spurious associations.

What to do when you cannot randomise#

An experiment is the answer when one is available. Module seven covers what to do when it is not, but the short version:

Stratify. Compute the effect within levels of the confounder. Straightforward and effective when you know the confounder and it has few levels.

Control for it in a regression. Include the confounder as a covariate. Works only for confounders you have measured and modelled with roughly the right functional form.

Match. Pair treated units with untreated units that look alike on observed characteristics, and compare within pairs.

Difference-in-differences. When a change hits one group and not another, compare the change over time between groups. Differencing removes any time-invariant confounder, which is a large class.

Every one of these handles only the confounders you thought of and measured. That is their shared limitation and the reason a randomised experiment, when possible, is worth an enormous amount.

Language discipline#

The words you use become the words in the summary that becomes the words in the decision. Use them precisely:

• ✗ "Feature X increases retention by 12%"
• ✓ "Users of feature X retain 12 points better. They also had 3× the prior engagement, so most of this gap is probably selection."

The second sentence is barely longer, and it is the difference between a stakeholder making a good decision and a bad one.