Field note · July 2, 2023

The overall average hides 14 different numbers

Real group means for one column across 14 segments, and what the pooled average conceals.

1 min read ·Analytics practice ·statistics practice

dest splits flight-delays into 14 groups. Here is what flight_no looks like inside each.

  • MSP — mean 3,748, median 3,770 (561 rows)
  • DFW — mean 3,641, median 3,622 (569 rows)
  • SEA — mean 3,635, median 3,657 (557 rows)
  • IAH — mean 3,627, median 3,658 (578 rows)
  • DEN — mean 3,619, median 3,736 (566 rows)
  • ATL — mean 3,571, median 3,651 (589 rows)

Top to bottom that is 3,748 against 3,391, a spread of 10.5%. The pooled average is 3,560.

sql
select dest,
  count(*)                                               as rows,
  round(avg(flight_no)::numeric, 2)                      as mean,
  percentile_cont(0.5) within group (order by flight_no) as median
from flight_delays
group by 1
order by mean desc;

The groups are close enough that the pooled average is a fair summary. That is worth confirming rather than assuming: the check costs one query and the failure mode is invisible.

Notice the mean and median columns disagree only slightly here. Always compute both in the group-by. The comparison between them per segment is free and tells you whether you are looking at a level difference or a tail difference.

This is the setup for Simpson's paradox — the case where every segment moves one way and the total moves the other.