Blog, page 35

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,201 posts. , or browse by year and tag below.

May 15, 2024

Counting rows is not testing grain

12 distinct combinations against 8,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

May 14, 2024

Before you clip the outliers

A max 1.6× the 99th percentile, and the three different things that could be.

statistics quality Statistics

May 13, 2024

What sched_dep_hour actually contains in flight-delays

8,000 rows, 17 distinct values, and the one fact about sched_dep_hour that changes how you query it.

quality practice Data quality

May 12, 2024

A essentially none correlation, and what it is not

A measured correlation of 0.04 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

May 11, 2024

The top of fare_usd is 2.4× the 99th percentile

A max 2.4× the 99th percentile, and the three different things that could be.

statistics quality Statistics

May 10, 2024

Five minutes with genre in movie-ratings

9,000 rows, 8 distinct values, and the one fact about genre that changes how you query it.

quality practice Data quality

May 9, 2024

The overall average hides 5 different numbers

Real group means for one column across 5 segments, and what the pooled average conceals.

statistics practice Analytics practice

May 8, 2024

Course note — Facts, dimensions, and picking a grain

Dimensional modelling is thirty years old and still the right default, because it optimises for the thing that actually matters — an analyst being able to answer a question without asking anyone.

practice Analytics engineering

May 7, 2024

r = 0.73 between distance_km and fare_usd

A measured correlation of 0.73 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

May 6, 2024

Reading genre before trusting it in movie-ratings

9,000 rows, 8 distinct values, and the one fact about genre that changes how you query it.

quality practice Data quality

May 5, 2024

Counting rows is not testing grain

900 distinct combinations against 900 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

May 4, 2024

support-tickets.priority: 4 values, 92.6% in the top three

4 values with 48.4% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

May 3, 2024

Reading company_size before trusting it in data-job-postings

3,500 rows, 4 distinct values, and the one fact about company_size that changes how you query it.

quality practice Data quality

May 2, 2024

Is trip_id + payment_type the grain of ride-hail-trips?

8,000 distinct combinations against 8,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

May 1, 2024

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Apr 30, 2024

Reading region before trusting it in saas-subscriptions

2,500 rows, 4 distinct values, and the one fact about region that changes how you query it.

quality practice Data quality

Apr 29, 2024

retail-orders.channel: 4 values, 83.6% in the top three

4 values with 39.1% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Apr 28, 2024

The top of distance_km is 1.7× the 99th percentile

A max 1.7× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Apr 27, 2024

Five minutes with country in data-job-postings

3,500 rows, 7 distinct values, and the one fact about country that changes how you query it.

quality practice Data quality

Apr 26, 2024

Pattern: The grain uniqueness test

One test per model, on the key that defines its grain. The cheapest quality control available and the one that catches the worst bugs.

quality Data quality

Apr 25, 2024

The overall average hides 6 different numbers

Real group means for one column across 6 segments, and what the pooled average conceals.

statistics practice Analytics practice

Apr 24, 2024

Reading fare_usd before trusting it in ride-hail-trips

8,000 rows, 3,430 distinct values, and the one fact about fare_usd that changes how you query it.

quality practice Data quality

Apr 23, 2024

The `else` branch that ate 4% of revenue

A new payment type, a forgiving CASE statement, and six weeks of quietly wrong numbers.

quality sql Data quality

Apr 22, 2024

Cutting a lesson down — Monitoring data, not just jobs

Job monitoring tells you the code ran. Data monitoring tells you the numbers are right. Teams that only have the first find out about the second from a stakeholder.

practice Data platform