Blog, page 33

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,201 posts. , or browse by year and tag below.

Jul 2, 2024

Fan-out join, in short

A join to a table with more than one row per key multiplies every measure on the other side. Nothing errors, and the number is plausible.

sql Analytics engineering

Jul 1, 2024

clinical-trial.site: 6 values, 52.7% in the top three

6 values with 19.3% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Jun 30, 2024

Before you clip the outliers

A max 4.1× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Jun 29, 2024

Five minutes with release_year in movie-ratings

9,000 rows, 53 distinct values, and the one fact about release_year that changes how you query it.

quality practice Data quality

Jun 28, 2024

Charting distance_km by payment_type

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Jun 27, 2024

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 26, 2024

What reading_date actually contains in city-air-quality

5,480 rows, 1,370 distinct values, and the one fact about reading_date that changes how you query it.

quality practice Data quality

Jun 25, 2024

The missing values were the finding

18% of salary fields were blank, and the blanks were not random.

statistics quality Statistics

Jun 24, 2024

r = 0.77 between gdp_per_capita_usd and life_expectancy

A measured correlation of 0.77 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 23, 2024

fare_usd by borough: a 229% spread

Real group means for one column across 5 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 22, 2024

What discount_pct actually contains in retail-orders

7,000 rows, 42 distinct values, and the one fact about discount_pct that changes how you query it.

quality practice Data quality

Jun 21, 2024

Reader question — The five tests that cover most questions

Hypothesis testing is taught as a zoo of named procedures. In practice a small number cover nearly everything, and choosing correctly is easier than remembering the zoo.

practice Experimentation

Jun 20, 2024

Contract check at the boundary, in short

Validate the shape of incoming data where it enters your system — fail loudly on structure, warn on distribution.

quality pipelines Data quality

Jun 19, 2024

What passengers actually contains in ride-hail-trips

8,000 rows, 6 distinct values, and the one fact about passengers that changes how you query it.

quality practice Data quality

Jun 18, 2024

saas-subscriptions.channel: 6 values, 60.9% in the top three

6 values with 26.4% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Jun 17, 2024

Is movie_id + rated_at the grain of movie-ratings?

9,000 distinct combinations against 9,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Jun 16, 2024

Before you clip the outliers

A max 1.2× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Jun 15, 2024

A essentially none correlation, and what it is not

A measured correlation of 0.09 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 14, 2024

Five minutes with mrr_usd in saas-subscriptions

2,500 rows, 2,269 distinct values, and the one fact about mrr_usd that changes how you query it.

quality practice Data quality

Jun 13, 2024

Charting humidity_pct by site

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Jun 12, 2024

Reader question — Features, and explaining what the model did

Feature engineering is where domain knowledge enters a model, and interpretation is where it leaves. Both are more valuable than algorithm selection.

practice Machine learning

Jun 11, 2024

Five minutes with converted in ab-test-checkout

6,000 rows, 2 distinct values, and the one fact about converted that changes how you query it.

quality practice Data quality

Jun 10, 2024

Baseline before model, in short

Build the dumbest thing that could work and measure it properly. Your model must beat it on the same split, or it has done nothing.

modelling Machine learning

Jun 9, 2024

Before you clip the outliers

A max 1.2× the 99th percentile, and the three different things that could be.

statistics quality Statistics