Field note · October 28, 2024

The top of revenue_usd is 3.0× the 99th percentile

A max 3.0× the 99th percentile, and the three different things that could be.

1 min read ·Statistics ·statistics quality

revenue_usd on retail-orders: p99 is 780 and the max is 2,318.

The max is 3.0× the 99th percentile and 14.5 standard deviations above the mean. The median is 60, so the top of this column lives in a different regime from the middle of it.

sql
select *
from retail_orders
order by revenue_usd desc
limit 20;

Run that before deciding anything. There are three things an extreme value can be, and they need opposite treatment: a data error, a real but rare event, or a different population that got mixed into the same table. Only the first is safe to drop, and you cannot tell them apart from the summary statistics — you have to look at the rows.

A ratio this large usually means the third case. Something in this column is not the same kind of thing as the rest, and the honest fix is a filter with a stated reason rather than a percentile clip.

Clipping at the 99th percentile is a decision to discard your most interesting rows. Sometimes right. Never automatic, and never silent — if a number in a report was computed on clipped data, the report has to say so.