Blog, page 30

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

Sep 11, 2024

Anti-join, in short

Find rows with no match on the other side — and avoid the NOT IN trap that silently returns nothing.

sql Analytics engineering

Sep 10, 2024

Five minutes with rated_at in movie-ratings

9,000 rows, 8,999 distinct values, and the one fact about rated_at that changes how you query it.

quality practice Data quality

Sep 9, 2024

Is order_id + category the grain of retail-orders?

6,355 distinct combinations against 7,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Sep 8, 2024

The overall average hides 6 different numbers

Real group means for one column across 6 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 7, 2024

Five minutes with remote in data-job-postings

3,500 rows, 3 distinct values, and the one fact about remote that changes how you query it.

quality practice Data quality

Sep 6, 2024

Before you clip the outliers

A max 1.3× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Sep 5, 2024

A essentially none correlation, and what it is not

A measured correlation of -0.09 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 4, 2024

Five minutes with bmi in clinical-trial

900 rows, 199 distinct values, and the one fact about bmi that changes how you query it.

quality practice Data quality

Sep 3, 2024

Charting seats_now by region

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Sep 2, 2024

Course note — Causal claims when you cannot randomise

Half the interesting questions cannot be tested. Here is what is available instead, what each method assumes, and how to state a conclusion you can defend.

practice Experimentation

Sep 1, 2024

Reading load_mw before trusting it in grid-energy-load

8,760 rows, 7,724 distinct values, and the one fact about load_mw that changes how you query it.

quality practice Data quality

Aug 31, 2024

The top of population is 1.1× the 99th percentile

A max 1.1× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Aug 30, 2024

The category that eats the chart

4 values with 46.5% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Aug 29, 2024

Reading pickup_at before trusting it in ride-hail-trips

8,000 rows, 7,998 distinct values, and the one fact about pickup_at that changes how you query it.

quality practice Data quality

Aug 28, 2024

The grain uniqueness test, in short

One test per model, on the key that defines its grain. The cheapest quality control available and the one that catches the worst bugs.

quality Data quality

Aug 27, 2024

A backfill that cost $400 in ninety seconds

Four hundred days, unbounded concurrency, and a very short conversation with the platform team.

engineering performance quality Data platform

Aug 26, 2024

The overall average hides 5 different numbers

Real group means for one column across 5 segments, and what the pooled average conceals.

statistics practice Analytics practice

Aug 25, 2024

What seats actually contains in saas-subscriptions

2,500 rows, 246 distinct values, and the one fact about seats that changes how you query it.

quality practice Data quality

Aug 24, 2024

Is user_id + device the grain of ab-test-checkout?

6,000 distinct combinations against 6,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Aug 23, 2024

r = 0.53 between fare_usd and surge_multiplier

A measured correlation of 0.53 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Aug 22, 2024

Five minutes with internet_pct in world-indicators

720 rows, 401 distinct values, and the one fact about internet_pct that changes how you query it.

quality practice Data quality

Aug 21, 2024

Charting dep_delay_min by carrier

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Aug 20, 2024

Course note — Monitoring data, not just jobs

Job monitoring tells you the code ran. Data monitoring tells you the numbers are right. Teams that only have the first find out about the second from a stakeholder.

practice Data platform

Aug 19, 2024

Sessionisation, in short

Turn a stream of timestamped events into sessions by flagging the gaps and cumulative-summing the flag.

sql Analytics engineering