Field note · July 20, 2023

Reading trip_id before trusting it in ride-hail-trips

8,000 rows, all distinct, and the one fact about trip_id that changes how you query it.

1 min read ·Data quality ·quality practice

Profiling trip_id on ride-hail-trips before anyone builds anything on top of it.

8,000 rows, no nulls, 8,000 distinct values. Stable synthetic identifier.

8,000 distinct values across 8,000 rows, so it is unique and a candidate key.

Uniqueness is a property of the data you have, not a guarantee about the data you will get. Nothing in the file enforces it; a re-export with an extra row, a merged upstream table, or a backfill that runs twice all break it, and none of them raise an error.

sql
select
  count(*)                  as rows,
  count(trip_id)            as present,
  count(*) - count(trip_id) as nulls,
  count(distinct trip_id)   as distinct_values
from ride_hail_trips;

Unique today is not the same as declared unique. If nothing tests it, the day it stops being unique is the day a join starts fanning out quietly.

Where this bites: the failure is silent in both directions. A duplicated key produces too many rows and a total that is too high; a missing one produces too few and a total that is too low. Neither errors, and both look like a real change to whoever reads the number.

The grain is one row per completed trip, which is the context every one of those numbers depends on. None of them survive a change of grain, which is why "profile the column" and "profile the table" are the same job. Full schema, and the CSV, on the dataset page.