Field note · July 13, 2023
The overall average hides 10 different numbers
Real group means for one column across 10 segments, and what the pooled average conceals.
city splits data-job-postings into 10 groups. Here is what salary_max_usd looks like inside each.
San Francisco— mean 176,026, median 174,250 (532 rows)New York— mean 168,826, median 172,500 (495 rows)Austin— mean 138,919, median 140,500 (253 rows)Chicago— mean 134,973, median 137,000 (299 rows)London— mean 85,831, median 92,000 (436 rows)Amsterdam— mean 83,441, median 92,500 (245 rows)
Top to bottom that is 176,026 against 33,385, a spread of 427.3%. The pooled average is 138,807.
select city,
count(*) as rows,
round(avg(salary_max_usd)::numeric, 2) as mean,
percentile_cont(0.5) within group (order by salary_max_usd) as median
from data_job_postings
group by 1
order by mean desc;A spread that wide means the pooled number is not a summary, it is an artefact of the mix. Change the proportion of San Francisco rows and the overall average moves without any individual group changing at all — which is how a metric goes up while every segment goes down.
Notice the mean and median columns disagree only slightly here. Always compute both in the group-by. The comparison between them per segment is free and tells you whether you are looking at a level difference or a tail difference.
This is the setup for Simpson's paradox — the case where every segment moves one way and the total moves the other.