Field note · November 15, 2024

Reading country before trusting it in retail-orders

7,000 rows, 8 distinct values, and the one fact about country that changes how you query it.

1 min read ·Data quality ·quality practice

Profiling country on retail-orders before anyone builds anything on top of it.

7,000 rows, no nulls, 8 distinct values. Two-letter shipping country code.

8 values, and they are not evenly spread: US 37.9%, GB 13.1%, DE 10.5%, CA 10.2%. The largest takes 37.9% on its own.

Read the distinct list rather than the distinct count. Casing differences and trailing whitespace produce values that look identical in a report and group separately in SQL, and the count will not show you that — select distinct country order by 1 will, in about a second.

sql
select
  count(*)                  as rows,
  count(country)            as present,
  count(*) - count(country) as nulls,
  count(distinct country)   as distinct_values
from retail_orders;

8 values is past the point where a bar chart stays readable. Group the tail explicitly rather than letting a chart library decide which 0 categories to drop for you.

Where this bites: a group-by on this column produces 8 rows today. If it is a column an upstream system can add values to, it produces an unknown number tomorrow, and any dashboard laid out for 8 categories reflows without warning. Values arriving is a schema change that no schema check catches.

The grain is one row per order line, which is the context every one of those numbers depends on. None of them survive a change of grain, which is why "profile the column" and "profile the table" are the same job. Full schema, and the CSV, on the dataset page.