Blog, page 43

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

Nov 4, 2023

Cutting a lesson down — When not to build a model

The most valuable modelling skill is recognising the problems that do not need one, which is a large majority of the problems people bring you.

practice Machine learning

Nov 3, 2023

Five minutes with status in sensor-telemetry

8,000 rows, 3 distinct values, and the one fact about status that changes how you query it.

quality practice Data quality

Nov 2, 2023

Is order_id + channel the grain of retail-orders?

4,169 distinct combinations against 7,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Nov 1, 2023

A essentially none correlation, and what it is not

A measured correlation of 0.07 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 31, 2023

Course note — Why your query costs what it costs

Columnar storage, partitioning, and file layout explain most of the difference between a query that returns in two seconds and the same query taking four minutes and forty dollars.

practice Data platform

Oct 30, 2023

Counting rows is not testing grain

8,000 distinct combinations against 8,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Oct 29, 2023

Reading csat before trusting it in support-tickets

4,000 rows, 5 distinct values, and the one fact about csat that changes how you query it.

quality practice Data quality

Oct 28, 2023

salary_max_usd by country: a 379% spread

Real group means for one column across 7 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 27, 2023

A weak correlation, and what it is not

A measured correlation of -0.37 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 26, 2023

Reader question — The anatomy of a silent failure

A walk through a real class of incident — the kind where every job is green, every test passes, and the number has been wrong for six weeks.

practice Data quality

Oct 25, 2023

Reading opened_at before trusting it in support-tickets

4,000 rows, 3,990 distinct values, and the one fact about opened_at that changes how you query it.

quality practice Data quality

Oct 24, 2023

Yesterday keeps changing and that is correct

Late-arriving events, a seven-day lookback, and how to tell finance about it without a crisis.

pipelines engineering Data engineering

Oct 23, 2023

Is line_no + country the grain of retail-orders?

32 distinct combinations against 7,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Oct 22, 2023

bmi by sex: a 1% spread

Real group means for one column across 2 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 21, 2023

Reading sched_dep_hour before trusting it in flight-delays

8,000 rows, 17 distinct values, and the one fact about sched_dep_hour that changes how you query it.

quality practice Data quality

Oct 20, 2023

r = 1.00 between salary_min_usd and salary_max_usd

A measured correlation of 1.00 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 19, 2023

Counting rows is not testing grain

2,500 distinct combinations against 2,500 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Oct 18, 2023

Five minutes with ticket_id in support-tickets

4,000 rows, all distinct, and the one fact about ticket_id that changes how you query it.

quality practice Data quality

Oct 17, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 16, 2023

r = -0.00 between bmi and week12_score

A measured correlation of -0.00 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 15, 2023

Five minutes with payment_type in ride-hail-trips

8,000 rows, 3 distinct values, and the one fact about payment_type that changes how you query it.

quality practice Data quality

Oct 14, 2023

The overall average hides 6 different numbers

Real group means for one column across 6 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 13, 2023

r = 0.02 between unit_price_usd and discount_pct

A measured correlation of 0.02 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 12, 2023

Five minutes with hour_at in grid-energy-load

8,760 rows, all distinct, and the one fact about hour_at that changes how you query it.

quality practice Data quality