Blog, page 5

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

May 4, 2026

The interval is the result

A point estimate of 2.1%, the interval around it, and what a tighter one would cost.

statistics Statistics

May 3, 2026

Is a data engineering certificate worth it?

For getting past a keyword filter, sometimes. For learning the job, rarely.

career Data engineering

May 2, 2026

Content is files, not a database

Every lesson, article, pattern and field note on this site is a markdown file in a git repository. No CMS, no database, no admin panel.

engineering practice Data platform

May 1, 2026

A five-minute use for Data Explorer

A fast table with per-column profiling: type inference, missing counts, distinct values, quantiles, and a sparkline distribution for every numeric column.

visualization Visualization

Apr 30, 2026

What is deliberately wrong with flight-delays

Eight thousand domestic flights across six carriers and fourteen airports, with departure and arrival delay, cancellations, and a delay distribution that is long-tailed rather than normal..

quality practice Data quality

Apr 29, 2026

Course note — Colour is an encoding, not decoration

Colour has exactly four jobs in a chart. Using it for anything else adds noise, and using it carelessly excludes about one in twelve of your readers.

practice Visualization

Apr 28, 2026

Pattern: Baseline before model

Build the dumbest thing that could work and measure it properly. Your model must beat it on the same split, or it has done nothing.

modelling Machine learning

Apr 27, 2026

signed_up_on runs 2023-01-04 to 2025-12-01

947 populated days across a 1,063-day span, and what that allows you to compute.

engineering quality Data engineering

Apr 26, 2026

The category that eats the chart

5 values with 33.4% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Apr 25, 2026

Is subject_id + arm the grain of clinical-trial?

900 distinct combinations against 900 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Apr 24, 2026

Finding the gaps in ab-test-checkout

gaps and islands against a real schema, and the clause in it that is doing the work.

sql Analytics engineering

Apr 23, 2026

user_id is not unique, and the join knows

8.2 rows per key on one table, and the totals that quietly inherit the multiplier.

sql quality Analytics engineering

Apr 22, 2026

183 zeros in arr_delay_min, and what they mean

2.3% of one column is exactly zero. Whether that is a measurement or an absence changes every aggregate built on it.

quality statistics Data quality

Apr 21, 2026

Check the blanks before you drop them

89 blanks in one column, and the one-line check that decides whether dropping them is free.

quality statistics Statistics

Apr 20, 2026

fare_usd: mean 22.16, median 14.29

A 55.1% gap between mean and median on one column, and which of the two answers the question you were asked.

statistics practice Statistics

Apr 19, 2026

Five minutes with year in world-indicators

720 rows, 24 distinct values, and the one fact about year that changes how you query it.

quality practice Data quality

Apr 18, 2026

The arithmetic behind a cheap query

Column pruning times partition pruning on a real schema, and the function call that quietly defeats both.

performance warehousing Data platform

Apr 17, 2026

Before you clip the outliers

A max 2.8× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Apr 16, 2026

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Apr 15, 2026

A strong correlation, and what it is not

A measured correlation of 0.73 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Apr 14, 2026

One question I ask in every code review

"What does one row of this represent?" — and what happens when nobody can answer.

sql warehousing practice Analytics engineering

Apr 13, 2026

Correction: Contract check at the boundary

The page claimed more than it should have. What it says now, and where the pattern stops applying.

quality practice Data quality

Apr 12, 2026

Is my pipeline actually idempotent?

There is one test. Run it twice on the same input and diff the output. If it differs, it is not.

pipelines engineering Data engineering

Apr 11, 2026

Charting sched_dep_hour by carrier

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization