Blog, page 27

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

Nov 22, 2024

Five minutes with humidity_pct in sensor-telemetry

8,000 rows, 260 distinct values, and the one fact about humidity_pct that changes how you query it.

quality practice Data quality

Nov 21, 2024

A strong correlation, and what it is not

A measured correlation of 0.94 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Nov 20, 2024

Charting csat by queue

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Nov 19, 2024

Cutting a lesson down — Writing SQL people can read, and reading the plan

The query you write twice a year is written for the machine. The one in a production model is written for the person debugging it at 3am, who might be you.

practice Analytics engineering

Nov 18, 2024

Reading humidity_pct before trusting it in sensor-telemetry

8,000 rows, 260 distinct values, and the one fact about humidity_pct that changes how you query it.

quality practice Data quality

Nov 17, 2024

Pattern: Difference-in-differences

When a change hits one group and not another, compare the change over time rather than the levels. Any time-invariant difference cancels out.

statistics experiments Experimentation

Nov 16, 2024

flight-delays.carrier: 6 values, 63.6% in the top three

6 values with 21.9% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Nov 15, 2024

Reading country before trusting it in retail-orders

7,000 rows, 8 distinct values, and the one fact about country that changes how you query it.

quality practice Data quality

Nov 14, 2024

Sessions out of raw events in ride-hail-trips

sessionisation against a real schema, and the clause in it that is doing the work.

sql Analytics engineering

Nov 13, 2024

Counting rows is not testing grain

6,000 distinct combinations against 6,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Nov 12, 2024

Five minutes with dropped_out in clinical-trial

900 rows, 2 distinct values, and the one fact about dropped_out that changes how you query it.

quality practice Data quality

Nov 11, 2024

The top of wind_mw is 1.6× the 99th percentile

A max 1.6× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Nov 10, 2024

The overall average hides 3 different numbers

Real group means for one column across 3 segments, and what the pooled average conceals.

statistics practice Analytics practice

Nov 9, 2024

What sessions actually contains in ab-test-checkout

6,000 rows, 20 distinct values, and the one fact about sessions that changes how you query it.

quality practice Data quality

Nov 8, 2024

r = 0.00 between no2_ppb and o3_ppb

A measured correlation of 0.00 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Nov 7, 2024

Charting bmi by site

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Nov 6, 2024

What resolved_at actually contains in support-tickets

4,000 rows, 3,775 distinct values, and the one fact about resolved_at that changes how you query it.

quality practice Data quality

Nov 5, 2024

data-job-postings.seniority: 5 values, 83.4% in the top three

5 values with 33.4% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Nov 4, 2024

Cutting a lesson down — Choosing the form before the colours

Most bad charts are bad because someone picked a chart type from a toolbar. The type should be decided by what the data is being asked to do.

practice Visualization

Nov 3, 2024

Five minutes with carrier in flight-delays

8,000 rows, 6 distinct values, and the one fact about carrier that changes how you query it.

quality practice Data quality

Nov 2, 2024

As-of feature computation, in short

Every feature must be computable from information available at prediction time. Write down each one's timestamp, and the leakage disappears.

modelling Machine learning

Nov 1, 2024

Sessions out of raw events in data-job-postings

sessionisation against a real schema, and the clause in it that is doing the work.

sql Analytics engineering

Oct 31, 2024

Five minutes with queue in support-tickets

4,000 rows, 5 distinct values, and the one fact about queue that changes how you query it.

quality practice Data quality

Oct 30, 2024

Counting rows is not testing grain

8,291 distinct combinations against 9,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering