Blog, page 36

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,201 posts. , or browse by year and tag below.

Apr 21, 2024

Pattern: Idempotent window replace

Every write replaces a named slice rather than adding to a pile, so rerunning a job is safe and backfilling is routine.

pipelines Data engineering

Apr 20, 2024

retail-orders.country: 8 values, 61.5% in the top three

8 values with 37.9% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Apr 19, 2024

What surge_multiplier actually contains in ride-hail-trips

8,000 rows, 24 distinct values, and the one fact about surge_multiplier that changes how you query it.

quality practice Data quality

Apr 18, 2024

r = 0.03 between humidity_pct and vibration_mm_s

A measured correlation of 0.03 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Apr 17, 2024

Cutting a lesson down — Correlation, confounding, and Simpson's paradox

The gap between "these move together" and "this causes that" is where most bad decisions live. Here is how to see the gap, and what to do when you cannot close it.

practice Statistics

Apr 16, 2024

What carrier actually contains in flight-delays

8,000 rows, 6 distinct values, and the one fact about carrier that changes how you query it.

quality practice Data quality

Apr 15, 2024

Is order_id + country the grain of retail-orders?

4,169 distinct combinations against 7,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Apr 14, 2024

The left join that became an inner join, in short

A predicate on the right table in WHERE silently discards the unmatched rows you wrote a left join to keep.

sql Analytics engineering

Apr 13, 2024

Before you clip the outliers

A max 3.0× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Apr 12, 2024

flight_no by origin: a 8% spread

Real group means for one column across 14 segments, and what the pooled average conceals.

statistics practice Analytics practice

Apr 11, 2024

What revenue_usd actually contains in retail-orders

7,000 rows, 5,719 distinct values, and the one fact about revenue_usd that changes how you query it.

quality practice Data quality

Apr 10, 2024

r = 0.00 between temp_c and rpm

A measured correlation of 0.00 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Apr 9, 2024

Course note — The first hour with an unfamiliar dataset

A repeatable routine for finding out what you are actually holding, before you compute anything anyone will act on.

practice Data quality

Apr 8, 2024

Pattern: The left join that became an inner join

A predicate on the right table in WHERE silently discards the unmatched rows you wrote a left join to keep.

sql Analytics engineering

Apr 7, 2024

The category that eats the chart

6 values with 24.2% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Apr 6, 2024

Is movie_id + user_id the grain of movie-ratings?

8,291 distinct combinations against 9,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Apr 5, 2024

Reading ticket_id before trusting it in support-tickets

4,000 rows, all distinct, and the one fact about ticket_id that changes how you query it.

quality practice Data quality

Apr 4, 2024

The top of salary_max_usd is 1.2× the 99th percentile

A max 1.2× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Apr 3, 2024

The overall average hides 2 different numbers

Real group means for one column across 2 segments, and what the pooled average conceals.

statistics practice Analytics practice

Apr 2, 2024

Five minutes with temp_c in sensor-telemetry

8,000 rows, 1,414 distinct values, and the one fact about temp_c that changes how you query it.

quality practice Data quality

Apr 1, 2024

r = 0.56 between duration_min and tip_usd

A measured correlation of 0.56 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Mar 31, 2024

Course note — Choosing the form before the colours

Most bad charts are bad because someone picked a chart type from a toolbar. The type should be decided by what the data is being asked to do.

practice Visualization

Mar 30, 2024

Baseline before model, in short

Build the dumbest thing that could work and measure it properly. Your model must beat it on the same split, or it has done nothing.

modelling Machine learning

Mar 29, 2024

Is device_id + site the grain of sensor-telemetry?

12 distinct combinations against 8,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering