Blog, page 50

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,244 posts. , or browse by year and tag below.

Jul 3, 2023

Five minutes with co2_tonnes_per_capita in world-indicators

720 rows, 686 distinct values, and the one fact about co2_tonnes_per_capita that changes how you query it.

quality practice Data quality

Jul 2, 2023

The overall average hides 14 different numbers

Real group means for one column across 14 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jul 1, 2023

Counting rows is not testing grain

8,000 distinct combinations against 8,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Jun 30, 2023

Course note — Owning something that runs without you

The last module, and the one about what you owe other people — documentation, runbooks, handover, and the decision to turn something off.

practice Data platform

Jun 29, 2023

population by region: a 63% spread

Real group means for one column across 5 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 28, 2023

r = 0.01 between no2_ppb and temp_c

A measured correlation of 0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 27, 2023

Schema drift, on a Friday, obviously

A vendor renamed a field. We found out from a stakeholder. Here is the boundary check we should have had.

pipelines quality Data engineering

Jun 26, 2023

Five minutes with salary_min_usd in data-job-postings

3,500 rows, 436 distinct values, and the one fact about salary_min_usd that changes how you query it.

quality practice Data quality

Jun 25, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 24, 2023

A weak correlation, and what it is not

A measured correlation of -0.25 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 23, 2023

Five minutes with gdp_per_capita_usd in world-indicators

720 rows, 719 distinct values, and the one fact about gdp_per_capita_usd that changes how you query it.

quality practice Data quality

Jun 22, 2023

Reader question — What a pipeline is really doing

Extract, load, transform — why the industry flipped the last two letters, and what a pipeline owes you beyond moving bytes.

practice Data engineering

Jun 21, 2023

The overall average hides 14 different numbers

Real group means for one column across 14 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 20, 2023

r = 0.08 between solar_mw and price_eur_mwh

A measured correlation of 0.08 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 19, 2023

Reading co2_tonnes_per_capita before trusting it in world-indicators

720 rows, 686 distinct values, and the one fact about co2_tonnes_per_capita that changes how you query it.

quality practice Data quality

Jun 18, 2023

pm25 by city: a 173% spread

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 17, 2023

r = 0.00 between flight_no and dep_delay_min

A measured correlation of 0.00 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 16, 2023

What account_id actually contains in saas-subscriptions

2,500 rows, all distinct, and the one fact about account_id that changes how you query it.

quality practice Data quality

Jun 15, 2023

Reader question — Testing data like you test code

Your transformation logic is tested by unit tests. The data flowing through it is not, and that is where the failures actually happen.

practice Data quality

Jun 14, 2023

r = -0.01 between age and baseline_score

A measured correlation of -0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jun 13, 2023

Five minutes with pm25 in city-air-quality

5,480 rows, 480 distinct values, and the one fact about pm25 that changes how you query it.

quality practice Data quality

Jun 12, 2023

Course note — Orchestration, and what "it runs every night" costs

The moment analysis becomes a scheduled job, a whole category of problems arrives: dependencies, retries, alerting, and someone's phone at 3am.

practice Data platform

Jun 11, 2023

The overall average hides 8 different numbers

Real group means for one column across 8 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jun 10, 2023

Reading origin before trusting it in flight-delays

8,000 rows, 14 distinct values, and the one fact about origin that changes how you query it.

quality practice Data quality