Describing a column without lying

Before any modelling, before any test: can you describe one column honestly? Most people cannot, because they reach for the mean automatically and the mean assumes a symmetry that money, duration, and count data almost never have.

The mean assumes symmetry it rarely has#

Open ride-hail-trips. Fares are lognormal — a long right tail of airport runs. The mean sits noticeably above the median. If you report "the average fare is X" to someone deciding on a pricing change, they will picture a typical trip costing X. Most trips cost less.

The rule is simple and widely ignored: for anything money-shaped, duration-shaped, or count-shaped, report the median, and report a high percentile beside it.

select
    percentile_cont(0.50) within group (order by fare_usd) as median,
    avg(fare_usd)                                          as mean,
    percentile_cont(0.90) within group (order by fare_usd) as p90,
    percentile_cont(0.99) within group (order by fare_usd) as p99
from ride_hail_trips;

The distance between median and mean is a skew diagnostic you get for free. If they are close, the mean is fine. If they are far apart, the mean is describing the tail rather than the typical case.

Spread, and why standard deviation misleads on skewed data#

Standard deviation is the natural companion to the mean and inherits its assumptions. On a lognormal fare column, "mean 14.20, sd 11.80" implies a range dipping below zero, which is impossible.

For skewed data, quote the interquartile range (p25 to p75) or simply the percentiles themselves. "Median 11.40, IQR 7.80–17.20, p99 62.00" is both accurate and immediately interpretable, and it takes no more space than the mean and sd.

The five-number habit#

For any numeric column, look at min, p25, median, p75, max — plus the count and the null count. Seven numbers, and they catch a remarkable share of problems:

• min < 0 on a quantity that cannot be negative → a data error, or a refund encoded as a negative.
• max wildly beyond p99 → an outlier that will dominate any mean, and possibly a unit error.
• min = max → a constant column.
• null count large → go read the missingness lesson before continuing.

Outliers are a decision, not a fact#

There is no statistical test that tells you a point is "wrong". Deciding what to do with extreme values is a judgement about the world, and it must be made explicitly.

Genuine extreme values — a real 62km airport trip. Keep them. Removing them is falsifying your data. If they dominate a mean, that is an argument for reporting the median, not for deleting rows.

Data errors — a fare of 4,000,000 because of a units bug. Remove them, log how many, and say so in the write-up.

A different population — the trips with distance_km = 0 in this dataset are billed cancellations, not trips. They should be analysed separately, not cleaned away and not averaged in.

The 1.5×IQR rule and the ±3 standard deviation rule are conventions, not laws. On lognormal data the IQR rule flags a large fraction of perfectly legitimate rows, because the distribution genuinely has a long tail. Applying it mechanically to skewed data deletes real signal.

Proportions need denominators#

"Conversion is 12%" is not a finding. 12% of 8 users is noise; 12% of 80,000 is a fact. Always publish the denominator, and for anything you might act on, the interval.

A serviceable interval for a proportion, avoiding the normal approximation's failure at small n or extreme p, is the Wilson interval:

from math import sqrt

def wilson(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """Confidence interval for a proportion that behaves sensibly at small n."""
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    denom = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return (max(0.0, centre - half), min(1.0, centre + half))

wilson(1, 8)      # (0.02, 0.47)  -- 12.5%, and essentially uninformative
wilson(10_000, 80_000)  # (0.1228, 0.1272)

Same 12.5%. Completely different findings.

Shape, in one glance#

Before summarising, plot the distribution. It takes ten seconds and it tells you which summary is legitimate.

• Bell-shaped → mean and sd are fine.
• Right-skewed → median; consider a log scale.
• Bimodal → stop. A bimodal column usually means two populations mixed together, and every summary of the combination describes neither. Find the variable that splits them and report separately. Trip duration in this dataset is mildly bimodal by rush hour; that is a finding, not a nuisance.
• Spike at zero → a zero-inflated distribution. Two processes: whether it happens, and how much given that it happened. Model and report them separately.

The Distribution Lab exists to build this intuition — sample from each shape and watch what the mean does.

Reporting a number to a human#

Match precision to certainty. "Revenue was $4,182,394.22" implies a precision you do not have when the underlying data restates for a week. "$4.18M" is more honest and more readable.

Round to the number of digits that could change your reader's decision — usually two or three significant figures. And a number without a comparison is not information: "conversion was 3.1%, against 2.8% last quarter" is a finding; "conversion was 3.1%" is a fact awaiting one.