Blog, page 46

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,245 posts. , or browse by year and tag below.

Oct 8, 2023

flight_no by carrier: a 3% spread

Real group means for one column across 6 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 7, 2023

A strong correlation, and what it is not

A measured correlation of 0.98 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 6, 2023

Reading passengers before trusting it in ride-hail-trips

8,000 rows, 6 distinct values, and the one fact about passengers that changes how you query it.

quality practice Data quality

Oct 5, 2023

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 4, 2023

A weak correlation, and what it is not

A measured correlation of -0.36 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Oct 3, 2023

Reading seniority before trusting it in data-job-postings

3,500 rows, 5 distinct values, and the one fact about seniority that changes how you query it.

quality practice Data quality

Oct 2, 2023

The overall average hides 2 different numbers

Real group means for one column across 2 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 1, 2023

A essentially none correlation, and what it is not

A measured correlation of 0.00 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 30, 2023

What exposed_on actually contains in ab-test-checkout

6,000 rows, 28 distinct values, and the one fact about exposed_on that changes how you query it.

quality practice Data quality

Sep 29, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 28, 2023

r = 0.80 between duration_min and fare_usd

A measured correlation of 0.80 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 27, 2023

Reading site before trusting it in sensor-telemetry

8,000 rows, 3 distinct values, and the one fact about site that changes how you query it.

quality practice Data quality

Sep 26, 2023

Reader question — The first hour with an unfamiliar dataset

A repeatable routine for finding out what you are actually holding, before you compute anything anyone will act on.

practice Data quality

Sep 25, 2023

r = -0.01 between bmi and baseline_score

A measured correlation of -0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 24, 2023

Five minutes with industry in saas-subscriptions

2,500 rows, 8 distinct values, and the one fact about industry that changes how you query it.

quality practice Data quality

Sep 23, 2023

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 22, 2023

r = 0.01 between dep_delay_min and distance_mi

A measured correlation of 0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 21, 2023

What adverse_event actually contains in clinical-trial

900 rows, 4 distinct values, and the one fact about adverse_event that changes how you query it.

quality practice Data quality

Sep 20, 2023

Cutting a lesson down — The five tests that cover most questions

Hypothesis testing is taught as a zoo of named procedures. In practice a small number cover nearly everything, and choosing correctly is easier than remembering the zoo.

practice Experimentation

Sep 19, 2023

DuckDB replaced a Spark cluster for one of our jobs

40 minutes to 90 seconds, and a lesson about what "big data" actually means.

engineering performance Data platform

Sep 18, 2023

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 17, 2023

r = -0.41 between pm25 and temp_c

A measured correlation of -0.41 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 16, 2023

Five minutes with week12_score in clinical-trial

900 rows, 434 distinct values, and the one fact about week12_score that changes how you query it.

quality practice Data quality

Sep 15, 2023

The overall average hides 30 different numbers

Real group means for one column across 30 segments, and what the pooled average conceals.

statistics practice Analytics practice