Field note · June 21, 2023
The overall average hides 14 different numbers
Real group means for one column across 14 segments, and what the pooled average conceals.
origin splits flight-delays into 14 groups. Here is what dep_delay_min looks like inside each.
JFK— mean 15, median 9 (548 rows)SFO— mean 14, median 9 (551 rows)ORD— mean 14, median 9 (560 rows)LAX— mean 11, median 6 (598 rows)ATL— mean 10, median 4 (600 rows)MSP— mean 9, median 4 (567 rows)
Top to bottom that is 15 against 7, a spread of 112.6%. The pooled average is 10.
select origin,
count(*) as rows,
round(avg(dep_delay_min)::numeric, 2) as mean,
percentile_cont(0.5) within group (order by dep_delay_min) as median
from flight_delays
group by 1
order by mean desc;A spread that wide means the pooled number is not a summary, it is an artefact of the mix. Change the proportion of JFK rows and the overall average moves without any individual group changing at all — which is how a metric goes up while every segment goes down.
Notice the mean and median columns disagree most in JFK, where the mean sits well above the median. Always compute both in the group-by. The comparison between them per segment is free and tells you whether you are looking at a level difference or a tail difference.
This is the setup for Simpson's paradox — the case where every segment moves one way and the total moves the other.