Blog, page 28

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

Oct 29, 2024

Two more tools, and why the palette is not our taste

The correlation explorer and regression lab are up. The colours were chosen by a script.

visualization Visualization

Oct 28, 2024

The top of revenue_usd is 3.0× the 99th percentile

A max 3.0× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Oct 27, 2024

What tip_usd actually contains in ride-hail-trips

8,000 rows, 1,276 distinct values, and the one fact about tip_usd that changes how you query it.

quality practice Data quality

Oct 26, 2024

no2_ppb by city: a 129% spread

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 25, 2024

A weak correlation, and what it is not

A measured correlation of 0.19 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 24, 2024

Five minutes with solar_mw in grid-energy-load

8,760 rows, 3,538 distinct values, and the one fact about solar_mw that changes how you query it.

quality practice Data quality

Oct 23, 2024

Charting seats by region

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Oct 22, 2024

The category that eats the chart

4 values with 39.1% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Oct 21, 2024

What posting_id actually contains in data-job-postings

3,500 rows, all distinct, and the one fact about posting_id that changes how you query it.

quality practice Data quality

Oct 20, 2024

Reader question — Orchestration, and what "it runs every night" costs

The moment analysis becomes a scheduled job, a whole category of problems arrives: dependencies, retries, alerting, and someone's phone at 3am.

practice Data platform

Oct 19, 2024

Gaps and islands, in short

Group consecutive runs — of dates, of statuses, of anything — by subtracting a row number from the value.

sql Analytics engineering

Oct 18, 2024

What revenue_usd actually contains in ab-test-checkout

6,000 rows, 603 distinct values, and the one fact about revenue_usd that changes how you query it.

quality practice Data quality

Oct 17, 2024

Finding the gaps in retail-orders

gaps and islands against a real schema, and the clause in it that is doing the work.

sql Analytics engineering

Oct 16, 2024

Counting rows is not testing grain

3,500 distinct combinations against 3,500 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Oct 15, 2024

What is_holiday actually contains in grid-energy-load

8,760 rows, 2 distinct values, and the one fact about is_holiday that changes how you query it.

quality practice Data quality

Oct 14, 2024

r = 0.19 between year and population

A measured correlation of 0.19 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 13, 2024

Before you clip the outliers

A max 1.1× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Oct 12, 2024

Reading pm10 before trusting it in city-air-quality

5,480 rows, 861 distinct values, and the one fact about pm10 that changes how you query it.

quality practice Data quality

Oct 11, 2024

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 10, 2024

Charting population by region

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Oct 9, 2024

Counting rows is not testing grain

3,500 distinct combinations against 3,500 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Oct 8, 2024

Reading pm25 before trusting it in city-air-quality

5,480 rows, 480 distinct values, and the one fact about pm25 that changes how you query it.

quality practice Data quality

Oct 7, 2024

Cutting a lesson down — Why your query costs what it costs

Columnar storage, partitioning, and file layout explain most of the difference between a query that returns in two seconds and the same query taking four minutes and forty dollars.

practice Data platform

Oct 6, 2024

Pattern: Lookback window

Reprocess the last N days every night so late-arriving events are picked up instead of landing in a closed window.

pipelines Data engineering