Blog, page 47

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,245 posts. , or browse by year and tag below.

Sep 14, 2023

Counting rows is not testing grain

3,500 distinct combinations against 3,500 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Sep 13, 2023

Five minutes with seats_now in saas-subscriptions

2,500 rows, 294 distinct values, and the one fact about seats_now that changes how you query it.

quality practice Data quality

Sep 12, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 11, 2023

r = -0.05 between population and gdp_per_capita_usd

A measured correlation of -0.05 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 10, 2023

What rating actually contains in movie-ratings

9,000 rows, 10 distinct values, and the one fact about rating that changes how you query it.

quality practice Data quality

Sep 9, 2023

Reader question — One metric, one definition

The reason three dashboards show three different revenue numbers is never a bug. It is three people who each wrote their own definition, and all of them were reasonable.

practice Analytics engineering

Sep 8, 2023

A essentially none correlation, and what it is not

A measured correlation of 0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 7, 2023

Five minutes with ordered_at in retail-orders

7,000 rows, 4,159 distinct values, and the one fact about ordered_at that changes how you query it.

quality practice Data quality

Sep 6, 2023

age by arm: a 1% spread

Real group means for one column across 2 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 5, 2023

r = -0.01 between sched_dep_hour and distance_mi

A measured correlation of -0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 4, 2023

Reading temp_c before trusting it in sensor-telemetry

8,000 rows, 1,414 distinct values, and the one fact about temp_c that changes how you query it.

quality practice Data quality

Sep 3, 2023

Counting rows is not testing grain

8,291 distinct combinations against 9,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Sep 2, 2023

The overall average hides 5 different numbers

Real group means for one column across 5 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 1, 2023

Reading tip_usd before trusting it in ride-hail-trips

8,000 rows, 1,276 distinct values, and the one fact about tip_usd that changes how you query it.

quality practice Data quality

Aug 31, 2023

r = -0.01 between flight_no and distance_mi

A measured correlation of -0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Aug 30, 2023

The overall average hides 8 different numbers

Real group means for one column across 8 segments, and what the pooled average conceals.

statistics practice Analytics practice

Aug 29, 2023

Reader question — Batch, streaming, and the honest difference

Streaming is not "batch but faster". It is a different consistency model with different failure modes, and most teams that adopt it did not need to.

practice Data engineering

Aug 28, 2023

A essentially none correlation, and what it is not

A measured correlation of -0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Aug 27, 2023

revenue_usd by device: a 28% spread

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Aug 26, 2023

Reading cancelled before trusting it in flight-delays

8,000 rows, 2 distinct values, and the one fact about cancelled that changes how you query it.

quality practice Data quality

Aug 25, 2023

r = -0.40 between pm10 and temp_c

A measured correlation of -0.40 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Aug 24, 2023

Cutting a lesson down — Validation that matches deployment

Cross-validation is not one technique. Choosing the wrong variant is how models get shipped with an offline score that has no relationship to what happens next.

practice Machine learning

Aug 23, 2023