Blog, page 31

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,201 posts. , or browse by year and tag below.

Aug 19, 2024

Sessionisation, in short

Turn a stream of timestamped events into sessions by flagging the gaps and cumulative-summing the flag.

sql Analytics engineering

Aug 18, 2024

The top of co2_tonnes_per_capita is 1.3× the 99th percentile

A max 1.3× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Aug 17, 2024

What mrr_usd actually contains in saas-subscriptions

2,500 rows, 2,269 distinct values, and the one fact about mrr_usd that changes how you query it.

quality practice Data quality

Aug 16, 2024

data-job-postings.company_size: 4 values, 81.1% in the top three

4 values with 29.6% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Aug 15, 2024

Counting rows is not testing grain

7,000 distinct combinations against 7,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Aug 14, 2024

Reading age before trusting it in clinical-trial

900 rows, 69 distinct values, and the one fact about age that changes how you query it.

quality practice Data quality

Aug 13, 2024

gdp_per_capita_usd by country: a 3,670% spread

Real group means for one column across 30 segments, and what the pooled average conceals.

statistics practice Analytics practice

Aug 12, 2024

A moderate correlation, and what it is not

A measured correlation of 0.48 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Aug 11, 2024

Reading flight_date before trusting it in flight-delays

8,000 rows, 365 distinct values, and the one fact about flight_date that changes how you query it.

quality practice Data quality

Aug 10, 2024

Charting mrr_usd by region

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Aug 9, 2024

Reader question — Schema drift, and the contract that prevents it

Your pipeline will break because someone upstream renamed a column on a Thursday and told nobody. Here is how to make that a caught error instead of a silent one.

practice Data engineering

Aug 8, 2024

Reading payment_type before trusting it in ride-hail-trips

8,000 rows, 3 distinct values, and the one fact about payment_type that changes how you query it.

quality practice Data quality

Aug 7, 2024

Slowly changing dimension, type 2, in short

Keep every version of a row so "what was true in February" stays answerable after March changes it.

warehousing modelling Analytics engineering

Aug 6, 2024

Is posting_id + city the grain of data-job-postings?

3,500 distinct combinations against 3,500 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Aug 5, 2024

Reading region before trusting it in world-indicators

720 rows, 5 distinct values, and the one fact about region that changes how you query it.

quality practice Data quality

Aug 4, 2024

seats_now by plan: a 25,267% spread

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Aug 3, 2024

The category that eats the chart

2 values with 50.0% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Aug 2, 2024

What remote actually contains in data-job-postings

3,500 rows, 3 distinct values, and the one fact about remote that changes how you query it.

quality practice Data quality

Aug 1, 2024

The top of sessions is 2.8× the 99th percentile

A max 2.8× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Jul 31, 2024

saas-subscriptions.industry: 8 values, 39.8% in the top three

8 values with 13.4% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Jul 30, 2024

Writing a SQL engine that fits in a browser tab

Tokeniser, Pratt parser, and the decision to be strict rather than forgiving.

sql engineering Visualization

Jul 29, 2024

Reading units before trusting it in retail-orders

7,000 rows, 5 distinct values, and the one fact about units that changes how you query it.

quality practice Data quality

Jul 28, 2024

A essentially none correlation, and what it is not

A measured correlation of 0.03 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jul 27, 2024

Charting age by site

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization