Five minutes with city in data-job-postings

Someone asked what is in city on data-job-postings, and the honest answer took one query.

3,500 rows, no nulls, 10 distinct values. Primary office city.

10 values, and they are not evenly spread: San Francisco 15.2%, New York 14.1%, London 12.5%, Bengaluru 11.9%. The largest takes 15.2% on its own.

Read the distinct list rather than the distinct count. Casing differences and trailing whitespace produce values that look identical in a report and group separately in SQL, and the count will not show you that — select distinct city order by 1 will, in about a second.

select
  count(*)               as rows,
  count(city)            as present,
  count(*) - count(city) as nulls,
  count(distinct city)   as distinct_values
from data_job_postings;

10 values is past the point where a bar chart stays readable. Group the tail explicitly rather than letting a chart library decide which 2 categories to drop for you.

Where this bites: a group-by on this column produces 10 rows today. If it is a column an upstream system can add values to, it produces an unknown number tomorrow, and any dashboard laid out for 10 categories reflows without warning. Values arriving is a schema change that no schema check catches.

The grain is one row per posting, which is the context every one of those numbers depends on. None of them survive a change of grain, which is why "profile the column" and "profile the table" are the same job. Full schema, and the CSV, on the dataset page.