Blog, page 44

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

Oct 11, 2023

salary_min_usd by seniority: a 162% spread

Real group means for one column across 5 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 10, 2023

A moderate correlation, and what it is not

A measured correlation of 0.70 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 9, 2023

Five minutes with pm10 in city-air-quality

5,480 rows, 861 distinct values, and the one fact about pm10 that changes how you query it.

quality practice Data quality

Oct 8, 2023

flight_no by carrier: a 3% spread

Real group means for one column across 6 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 7, 2023

A strong correlation, and what it is not

A measured correlation of 0.98 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 6, 2023

Reading passengers before trusting it in ride-hail-trips

8,000 rows, 6 distinct values, and the one fact about passengers that changes how you query it.

quality practice Data quality

Oct 5, 2023

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 4, 2023

A weak correlation, and what it is not

A measured correlation of -0.36 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 3, 2023

Reading seniority before trusting it in data-job-postings

3,500 rows, 5 distinct values, and the one fact about seniority that changes how you query it.

quality practice Data quality

Oct 2, 2023

The overall average hides 2 different numbers

Real group means for one column across 2 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 1, 2023

A essentially none correlation, and what it is not

A measured correlation of 0.00 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 30, 2023

What exposed_on actually contains in ab-test-checkout

6,000 rows, 28 distinct values, and the one fact about exposed_on that changes how you query it.

quality practice Data quality

Sep 29, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 28, 2023

r = 0.80 between duration_min and fare_usd

A measured correlation of 0.80 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 27, 2023

Reading site before trusting it in sensor-telemetry

8,000 rows, 3 distinct values, and the one fact about site that changes how you query it.

quality practice Data quality

Sep 26, 2023

Reader question — The first hour with an unfamiliar dataset

A repeatable routine for finding out what you are actually holding, before you compute anything anyone will act on.

practice Data quality

Sep 25, 2023

r = -0.01 between bmi and baseline_score

A measured correlation of -0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 24, 2023

Five minutes with industry in saas-subscriptions

2,500 rows, 8 distinct values, and the one fact about industry that changes how you query it.

quality practice Data quality

Sep 23, 2023

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 22, 2023

r = 0.01 between dep_delay_min and distance_mi

A measured correlation of 0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 21, 2023

What adverse_event actually contains in clinical-trial

900 rows, 4 distinct values, and the one fact about adverse_event that changes how you query it.

quality practice Data quality

Sep 20, 2023

Cutting a lesson down — The five tests that cover most questions

Hypothesis testing is taught as a zoo of named procedures. In practice a small number cover nearly everything, and choosing correctly is easier than remembering the zoo.

practice Experimentation

Sep 19, 2023

DuckDB replaced a Spark cluster for one of our jobs

40 minutes to 90 seconds, and a lesson about what "big data" actually means.

engineering performance Data platform

Sep 18, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice