Blog, page 20

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

May 9, 2025

Movie ratings, in one paragraph

Nine thousand user ratings over 260 films, with genres, release years, and the long-tail popularity distribution that makes naive recommenders look better than they are..

quality practice Data quality

May 8, 2025

Reader question — What a data team is actually for

The job is not producing numbers. It is reducing the number of decisions that get made badly, and almost everything else follows from that.

practice Analytics practice

May 7, 2025

Pattern: As-of feature computation

Every feature must be computable from information available at prediction time. Write down each one's timestamp, and the leakage disappears.

modelling Machine learning

May 6, 2025

The category that eats the chart

3 values with 34.8% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

May 5, 2025

Comparing a period to the one before it

period comparison against a real schema, and the clause in it that is doing the work.

sql Analytics engineering

May 4, 2025

Is flight_no + carrier the grain of flight-delays?

7,193 distinct combinations against 8,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

May 3, 2025

tip_usd: mean 3.13, median 1.96

A 59.6% gap between mean and median on one column, and which of the two answers the question you were asked.

statistics practice Statistics

May 2, 2025

Five minutes with category in retail-orders

7,000 rows, 6 distinct values, and the one fact about category that changes how you query it.

quality practice Data quality

May 1, 2025

Before you clip the outliers

A max 1.6× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Apr 30, 2025

The overall average hides 2 different numbers

Real group means for one column across 2 segments, and what the pooled average conceals.

statistics practice Analytics practice

Apr 29, 2025

The churn model that broke its own evaluation

A feedback loop, a holdout we should have kept, and the cost of finding out late.

modelling production quality Machine learning

Apr 28, 2025

r = 0.38 between tip_usd and surge_multiplier

A measured correlation of 0.38 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Apr 27, 2025

Charting temp_c by status

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Apr 26, 2025

Regression Lab works offline

Fit an ordinary least squares line to any pair of columns and get the diagnostics that decide whether the fit means anything: residual plot, R², standard errors, and leverage.

visualization Visualization

Apr 25, 2025

What is deliberately wrong with data-job-postings

Thirty-five hundred data job ads with titles, seniority, location, remote policy, salary bands, and a pipe-delimited skills column that needs parsing before it is useful..

quality practice Data quality

Apr 24, 2025

Reader question — Idempotency, backfills, and yesterday arriving twice

The single property that separates a pipeline you can operate from one you have to babysit — and the operation that will test it every time.

practice Data engineering

Apr 23, 2025

The category that eats the chart

4 values with 25.0% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Apr 22, 2025

Deterministic surrogate key, in short

Derive the key from the content instead of an auto-increment, so reprocessing the same row produces the same key.

pipelines modelling Data engineering

Apr 21, 2025

What flight_no actually contains in flight-delays

8,000 rows, 4,715 distinct values, and the one fact about flight_no that changes how you query it.

quality practice Data quality

Apr 20, 2025

Is posting_id + seniority the grain of data-job-postings?

3,500 distinct combinations against 3,500 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Apr 19, 2025

Finding the gaps in city-air-quality

gaps and islands against a real schema, and the clause in it that is doing the work.

sql Analytics engineering

Apr 18, 2025

co2_tonnes_per_capita: mean 3.67, median 2.80

A 31.1% gap between mean and median on one column, and which of the two answers the question you were asked.

statistics practice Statistics

Apr 17, 2025

The top of dep_delay_min is 3.1× the 99th percentile

A max 3.1× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Apr 16, 2025

The overall average hides 5 different numbers

Real group means for one column across 5 segments, and what the pooled average conceals.

statistics practice Analytics practice