Blog, page 49

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

Jun 13, 2023

Five minutes with pm25 in city-air-quality

5,480 rows, 480 distinct values, and the one fact about pm25 that changes how you query it.

quality practice Data quality

Jun 12, 2023

Course note — Orchestration, and what "it runs every night" costs

The moment analysis becomes a scheduled job, a whole category of problems arrives: dependencies, retries, alerting, and someone's phone at 3am.

practice Data platform

Jun 11, 2023

The overall average hides 8 different numbers

Real group means for one column across 8 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 10, 2023

Reading origin before trusting it in flight-delays

8,000 rows, 14 distinct values, and the one fact about origin that changes how you query it.

quality practice Data quality

Jun 9, 2023

A moderate correlation, and what it is not

A measured correlation of 0.47 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 8, 2023

Reader question — Why your query costs what it costs

Columnar storage, partitioning, and file layout explain most of the difference between a query that returns in two seconds and the same query taking four minutes and forty dollars.

practice Data platform

Jun 7, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 6, 2023

A essentially none correlation, and what it is not

A measured correlation of 0.00 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 5, 2023

Reading priority before trusting it in support-tickets

4,000 rows, 4 distinct values, and the one fact about priority that changes how you query it.

quality practice Data quality

Jun 4, 2023

Course note — Correlation, confounding, and Simpson's paradox

The gap between "these move together" and "this causes that" is where most bad decisions live. Here is how to see the gap, and what to do when you cannot close it.

practice Statistics

Jun 3, 2023

revenue_usd by country: a 12% spread

Real group means for one column across 8 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 2, 2023

r = -0.36 between pm25 and o3_ppb

A measured correlation of -0.36 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 1, 2023

What csat actually contains in support-tickets

4,000 rows, 5 distinct values, and the one fact about csat that changes how you query it.

quality practice Data quality

May 31, 2023

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

May 30, 2023

The meeting where the median won

Two years of planning around an average that described nobody.

statistics practice Statistics

May 29, 2023

Course note — Slowly changing dimensions, and the "as of when" problem

A customer moved from Berlin to Madrid last March. Was last February's revenue German or Spanish? The answer is a design decision, and pretending otherwise is how reports stop reconciling.

practice Analytics engineering

May 28, 2023

Five minutes with reading_date in city-air-quality

5,480 rows, 1,370 distinct values, and the one fact about reading_date that changes how you query it.

quality practice Data quality

May 27, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

May 26, 2023

A strong correlation, and what it is not

A measured correlation of 0.72 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

May 25, 2023

Five minutes with resolved_at in support-tickets

4,000 rows, 3,775 distinct values, and the one fact about resolved_at that changes how you query it.

quality practice Data quality

May 24, 2023

Reader question — Monitoring data, not just jobs

Job monitoring tells you the code ran. Data monitoring tells you the numbers are right. Teams that only have the first find out about the second from a stakeholder.

practice Data platform

May 23, 2023