Blog, page 37

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,201 posts. , or browse by year and tag below.

Mar 28, 2024

Before you clip the outliers

A max 1.2× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Mar 27, 2024

Reading is_returning before trusting it in ab-test-checkout

6,000 rows, 2 distinct values, and the one fact about is_returning that changes how you query it.

quality practice Data quality

Mar 26, 2024

first_response_min by priority: a 1,095% spread

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Mar 25, 2024

r = 0.98 between seats and seats_now

A measured correlation of 0.98 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Mar 24, 2024

What temp_c actually contains in grid-energy-load

8,760 rows, 395 distinct values, and the one fact about temp_c that changes how you query it.

quality practice Data quality

Mar 23, 2024

Reader question — Correlation, confounding, and Simpson's paradox

The gap between "these move together" and "this causes that" is where most bad decisions live. Here is how to see the gap, and what to do when you cannot close it.

practice Statistics

Mar 22, 2024

Bootstrap any statistic, in short

When you cannot remember the standard error formula — or there is not one — resample. It works for medians, percentiles and ratios alike.

statistics Statistics

Mar 21, 2024

What ticket_id actually contains in support-tickets

4,000 rows, all distinct, and the one fact about ticket_id that changes how you query it.

quality practice Data quality

Mar 20, 2024

bmi by site: a 3% spread

Real group means for one column across 6 segments, and what the pooled average conceals.

statistics practice Analytics practice

Mar 19, 2024

The chart builder is live, and it refuses to do things

What we deliberately made impossible, and the argument we had about one of them.

visualization Visualization

Mar 18, 2024

Is user_id + rated_at the grain of movie-ratings?

9,000 distinct combinations against 9,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Mar 17, 2024

Five minutes with reading_at in sensor-telemetry

8,000 rows, all distinct, and the one fact about reading_at that changes how you query it.

quality practice Data quality

Mar 16, 2024

The top of seats_now is 1.5× the 99th percentile

A max 1.5× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Mar 15, 2024

r = 0.01 between age and week12_score

A measured correlation of 0.01 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Mar 14, 2024

What pickup_at actually contains in ride-hail-trips

8,000 rows, 7,998 distinct values, and the one fact about pickup_at that changes how you query it.

quality practice Data quality

Mar 13, 2024

Cutting a lesson down — Slowly changing dimensions, and the "as of when" problem

A customer moved from Berlin to Madrid last March. Was last February's revenue German or Spanish? The answer is a design decision, and pretending otherwise is how reports stop reconciling.

practice Analytics engineering

Mar 12, 2024

Merge with a recency guard, in short

Upsert on a natural key, and refuse to let an out-of-order replay overwrite good data with stale data.

pipelines Data engineering

Mar 11, 2024

What genre actually contains in movie-ratings

9,000 rows, 8 distinct values, and the one fact about genre that changes how you query it.

quality practice Data quality

Mar 10, 2024

vibration_mm_s by status: a 191% spread

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Mar 9, 2024

Is flight_no + origin the grain of flight-delays?

7,690 distinct combinations against 8,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Mar 8, 2024

What company_size actually contains in data-job-postings

3,500 rows, 4 distinct values, and the one fact about company_size that changes how you query it.

quality practice Data quality

Mar 7, 2024

Before you clip the outliers

A max 1.7× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Mar 6, 2024

r = -0.02 between humidity_pct and rpm

A measured correlation of -0.02 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Mar 5, 2024

Reading year before trusting it in world-indicators

720 rows, 24 distinct values, and the one fact about year that changes how you query it.

quality practice Data quality