Field note · May 1, 2024
The overall average hides 3 different numbers
Real group means for one column across 3 segments, and what the pooled average conceals.
variant splits ab-test-checkout into 3 groups. Here is what sessions looks like inside each.
variant_a— mean 3, median 2 (1,937 rows)variant_b— mean 3, median 2 (2,089 rows)control— mean 3, median 2 (1,974 rows)
Top to bottom that is 3 against 3, a spread of 2.3%. The pooled average is 3.
select variant,
count(*) as rows,
round(avg(sessions)::numeric, 2) as mean,
percentile_cont(0.5) within group (order by sessions) as median
from ab_test_checkout
group by 1
order by mean desc;The groups are close enough that the pooled average is a fair summary. That is worth confirming rather than assuming: the check costs one query and the failure mode is invisible.
Notice the mean and median columns disagree most in variant_a, where the mean sits well above the median. Always compute both in the group-by. The comparison between them per segment is free and tells you whether you are looking at a level difference or a tail difference.
This is the setup for Simpson's paradox — the case where every segment moves one way and the total moves the other.