Causal claims when you cannot randomise

You cannot randomise a price change across a country. You cannot withhold a safety feature. You cannot randomise which customers get acquired through which channel. The decisions still have to be made.

Observational causal inference is what is left. It is genuinely useful and genuinely weaker than an experiment, and the honest practice of it is mostly about being explicit about assumptions you cannot verify.

Start with the causal question, written down#

Not "is X associated with Y" but: "if we intervened to set X, what would happen to Y, for whom, compared with what?"

Writing it that way forces four decisions: the treatment, the outcome, the population, and the counterfactual. Most confused analyses are confused because one of those four was never specified.

Draw the graph#

Sketch what you believe causes what. Boxes and arrows on paper is enough — the formalism is called a DAG, and the value comes from the drawing, not the theory.

   marketing spend ──▶ signups ──▶ revenue
          │                          ▲
          └────── seasonality ───────┘

The graph tells you what to control for. Control for confounders (common causes of treatment and outcome — seasonality here). Do not control for colliders (common effects), or for mediators if you want the total effect — controlling for signups here would remove most of the effect you are trying to measure.

That last point catches experienced people. "Control for everything you have" is wrong and actively harmful. Which variables to include is a question about your causal beliefs, not about model fit.

Difference-in-differences#

The workhorse. A change hits one group and not another; compare the change over time, not the levels.

                 before    after    change
treated group      100      130      +30
control group       90      105      +15
                                     ────
difference-in-differences            +15

Any time-invariant difference between the groups cancels out. That is a large class of confounders removed by arithmetic.

The assumption: parallel trends. Absent the treatment, both groups would have moved the same way. Unverifiable in principle — but testable in the past. Plot several pre-treatment periods. If the lines were not parallel before, the assumption is not credible and you should say so rather than proceed quietly.

Regression discontinuity#

When treatment is assigned by a threshold — a credit score above 700, a discount over $50, a free-shipping cutoff — units just either side of the cutoff are nearly identical except for treatment.

Strong design when it applies. Two things to check: units cannot manipulate their position (if they can nudge themselves over the line, the comparison breaks — look for a suspicious pile-up just above the threshold), and the estimate is local to the cutoff. An effect measured at score 700 says nothing about score 500.

Matching and propensity scores#

Pair each treated unit with untreated units that look similar on observed covariates, then compare within pairs.

The assumption is heavy: no unobserved confounders. You must have measured everything that drives both selection into treatment and the outcome. In business data, motivation, intent and prior engagement are usually the confounders and usually unmeasured.

Matching's real value is often diagnostic. If, after matching, the treated and control groups are still visibly different on the covariates you did measure, the naive comparison was even worse than you thought.

Instrumental variables#

Find something that affects treatment but affects the outcome only through treatment. Distance to a facility, an arbitrary policy rollout order, a random encouragement.

Powerful, and hard: good instruments are rare, and the exclusion restriction — no other path from instrument to outcome — is untestable and usually implausible on inspection. Weak instruments produce badly biased estimates with confident-looking standard errors, which is the worst combination available.

Synthetic control#

For a single treated unit — one country, one store, one market. Construct a weighted combination of untreated units that matches the treated one's pre-period trajectory, then compare post-treatment.

Excellent for geo experiments and policy changes. Requires a decent pre-period and a donor pool that plausibly spans the treated unit's behaviour.

The ladder of credibility#

Roughly, from strongest to weakest:

1. Randomised experiment
2. Natural experiment / regression discontinuity
3. Difference-in-differences with credible parallel pre-trends
4. Synthetic control
5. Matching on rich covariates
6. Regression with controls
7. Correlation with a story

Most business "analysis" lives at 6 and gets communicated as though it were at 1. Knowing where you are on this ladder is most of the discipline.

Sensitivity: how wrong could you be?#

Do not stop at the estimate. Ask what it would take to overturn it.

• How strong would an unmeasured confounder need to be to explain the whole effect? If a modest one would do it, your result is fragile — say so.
• Placebo test. Run the same analysis on a period or outcome where there should be no effect. If you find one, your method is picking up something other than the treatment.
• Vary the specification. Different control sets, different windows, different functional forms. If the estimate swings from +2% to +15%, report the range rather than your favourite point in it.