Field note · October 5, 2023

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

1 min read ·Analytics practice ·statistics practice

status splits sensor-telemetry into 3 groups. Here is what temp_c looks like inside each.

  • fault — mean 51.81, median 49.85 (124 rows)
  • warn — mean 45.55, median 45.53 (482 rows)
  • ok — mean 40.98, median 40.97 (7,394 rows)

Top to bottom that is 51.81 against 40.98, a spread of 26.4%. The pooled average is 41.42.

sql
select status,
  count(*)                                            as rows,
  round(avg(temp_c)::numeric, 2)                      as mean,
  percentile_cont(0.5) within group (order by temp_c) as median
from sensor_telemetry
group by 1
order by mean desc;

The groups are close enough that the pooled average is a fair summary. That is worth confirming rather than assuming: the check costs one query and the failure mode is invisible.

Notice the mean and median columns disagree only slightly here. Always compute both in the group-by. The comparison between them per segment is free and tells you whether you are looking at a level difference or a tail difference.

This is the setup for Simpson's paradox — the case where every segment moves one way and the total moves the other.