Field note · February 26, 2024

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

1 min read ·Analytics practice ·statistics practice

device splits ab-test-checkout into 3 groups. Here is what revenue_usd looks like inside each.

  • tablet — mean 7.87, median 0.00 (408 rows)
  • desktop — mean 7.84, median 0.00 (2,308 rows)
  • mobile — mean 6.16, median 0.00 (3,284 rows)

Top to bottom that is 7.87 against 6.16, a spread of 27.7%. The pooled average is 6.92.

sql
select device,
  count(*)                                                 as rows,
  round(avg(revenue_usd)::numeric, 2)                      as mean,
  percentile_cont(0.5) within group (order by revenue_usd) as median
from ab_test_checkout
group by 1
order by mean desc;

The groups are close enough that the pooled average is a fair summary. That is worth confirming rather than assuming: the check costs one query and the failure mode is invisible.

Notice the mean and median columns disagree most in tablet, where the mean sits well above the median. Always compute both in the group-by. The comparison between them per segment is free and tells you whether you are looking at a level difference or a tail difference.

This is the setup for Simpson's paradox — the case where every segment moves one way and the total moves the other.