Difference-in-differences

You meet it when#

You cannot randomise. A price changed in one market, a policy landed in one region, a feature rolled out to one platform first. The decision still has to be made.

The pattern#

                 before    after    change
treated group      100      130      +30
control group       90      105      +15
                                     ────
difference-in-differences            +15

The treated group rose 30. But everything was rising — the control rose 15 without any treatment. The attributable effect is the difference of the differences.

Any difference between the groups that is constant over time cancels out. That is a large and useful class of confounder removed by arithmetic rather than by modelling.

select
    avg(case when treated and period = 'after'  then y end)
  - avg(case when treated and period = 'before' then y end)
  - avg(case when not treated and period = 'after'  then y end)
  + avg(case when not treated and period = 'before' then y end) as did
from panel;

The assumption, and how to interrogate it#

Parallel trends: absent the treatment, both groups would have moved the same way.

This is unverifiable in principle — it is a claim about a world that did not happen. But it is testable in the past. Plot several pre-treatment periods for both groups. If the lines were not parallel before the treatment, there is no reason to believe they would have been after, and the estimate is not credible.

Publish that plot. It is the single most persuasive artefact in an observational analysis, and its absence is the tell that nobody checked.

Three checks worth running#

Placebo test. Run the same analysis on a period where nothing happened. If you find an effect, your method is picking up something other than the treatment.

Vary the control. Different comparison groups, different windows. If the estimate swings from +2% to +15%, report the range rather than your favourite point in it.

Sensitivity. How strong would an unmeasured, time-varying confounder need to be to explain the whole effect? If a modest one would do it, say so.