Blog, page 29

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,200 posts. , or browse by year and tag below.

Oct 5, 2024

The top of salary_min_usd is 1.2× the 99th percentile

A max 1.2× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Oct 4, 2024

pm10 by city: a 174% spread

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Oct 3, 2024

Reading is_holiday before trusting it in grid-energy-load

8,760 rows, 2 distinct values, and the one fact about is_holiday that changes how you query it.

quality practice Data quality

Oct 2, 2024

sensor-telemetry.site: 3 values, 100.0% in the top three

3 values with 35.3% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Oct 1, 2024

Finding the gaps in flight-delays

gaps and islands against a real schema, and the clause in it that is doing the work.

sql Analytics engineering

Sep 30, 2024

Five minutes with company_size in data-job-postings

3,500 rows, 4 distinct values, and the one fact about company_size that changes how you query it.

quality practice Data quality

Sep 29, 2024

A moderate correlation, and what it is not

A measured correlation of -0.41 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 28, 2024

Charting duration_min by payment_type

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Sep 27, 2024

The category that eats the chart

30 values with 3.3% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Sep 26, 2024

What city actually contains in data-job-postings

3,500 rows, 10 distinct values, and the one fact about city that changes how you query it.

quality practice Data quality

Sep 25, 2024

Course note — Idempotency, backfills, and yesterday arriving twice

The single property that separates a pipeline you can operate from one you have to babysit — and the operation that will test it every time.

practice Data engineering

Sep 24, 2024

We rewrote our interview loop and the hires got better

Less recall, more judgement, and a written scorecard for what each stage tests.

career practice Analytics practice

Sep 23, 2024

Difference-in-differences, in short

When a change hits one group and not another, compare the change over time rather than the levels. Any time-invariant difference cancels out.

statistics experiments Experimentation

Sep 22, 2024

Five minutes with region in world-indicators

720 rows, 5 distinct values, and the one fact about region that changes how you query it.

quality practice Data quality

Sep 21, 2024

Is reading_at + site the grain of sensor-telemetry?

8,000 distinct combinations against 8,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Sep 20, 2024

The top of price_eur_mwh is 1.2× the 99th percentile

A max 1.2× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Sep 19, 2024

Five minutes with fare_usd in ride-hail-trips

8,000 rows, 3,430 distinct values, and the one fact about fare_usd that changes how you query it.

quality practice Data quality

Sep 18, 2024

The overall average hides 4 different numbers

Real group means for one column across 4 segments, and what the pooled average conceals.

statistics practice Analytics practice

Sep 17, 2024

A strong correlation, and what it is not

A measured correlation of 0.75 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Sep 16, 2024

What wind_kph actually contains in city-air-quality

5,480 rows, 290 distinct values, and the one fact about wind_kph that changes how you query it.

quality practice Data quality

Sep 15, 2024

Charting salary_min_usd by seniority

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Sep 14, 2024

Reader question — Choosing the form before the colours

Most bad charts are bad because someone picked a chart type from a toolbar. The type should be decided by what the data is being asked to do.

practice Visualization

Sep 13, 2024

Five minutes with plan in saas-subscriptions

2,500 rows, 4 distinct values, and the one fact about plan that changes how you query it.

quality practice Data quality

Sep 12, 2024

clinical-trial.arm: 2 values, 100.0% in the top three

2 values with 50.0% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization