Field note · August 19, 2023

duration_min by borough: a 254% spread

Real group means for one column across 5 segments, and what the pooled average conceals.

1 min read ·Analytics practice ·statistics practice

Breaking duration_min down by borough on ride-hail-trips, because the headline average is 20.11 and no segment is actually there.

  • Airport — mean 53.77, median 48.50 (965 rows)
  • Uptown — mean 15.68, median 13.30 (1,453 rows)
  • Midtown — mean 15.62, median 13.20 (2,034 rows)
  • Downtown — mean 15.41, median 13.20 (2,399 rows)
  • Harbour — mean 15.20, median 13.30 (1,149 rows)

Top to bottom that is 53.77 against 15.20, a spread of 253.7%. The pooled average is 20.11.

sql
select borough,
  count(*)                                                  as rows,
  round(avg(duration_min)::numeric, 2)                      as mean,
  percentile_cont(0.5) within group (order by duration_min) as median
from ride_hail_trips
group by 1
order by mean desc;

A spread that wide means the pooled number is not a summary, it is an artefact of the mix. Change the proportion of Airport rows and the overall average moves without any individual group changing at all — which is how a metric goes up while every segment goes down.

Notice the mean and median columns disagree only slightly here. Always compute both in the group-by. The comparison between them per segment is free and tells you whether you are looking at a level difference or a tail difference.

This is the setup for Simpson's paradox — the case where every segment moves one way and the total moves the other.