Blog, page 32

Field notes — one a day since May 2023. Things we shipped, things we broke, and numbers we went and checked.

2026 2025 2024 2023

1,201 posts. , or browse by year and tag below.

Jul 26, 2024

Five minutes with seats in saas-subscriptions

2,500 rows, 246 distinct values, and the one fact about seats that changes how you query it.

quality practice Data quality

Jul 25, 2024

Reader question — A working setup that will not embarrass you

The minimum professional toolchain — version control, environments, notebooks used correctly, and the habits that make work reproducible six months later by someone who is not you.

practice Data engineering

Jul 24, 2024

Lookback window, in short

Reprocess the last N days every night so late-arriving events are picked up instead of landing in a closed window.

pipelines Data engineering

Jul 23, 2024

r = 0.07 between age and bmi

A measured correlation of 0.07 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jul 22, 2024

Reading plan before trusting it in saas-subscriptions

2,500 rows, 4 distinct values, and the one fact about plan that changes how you query it.

quality practice Data quality

Jul 21, 2024

Is line_no + channel the grain of retail-orders?

16 distinct combinations against 7,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Jul 20, 2024

Before you clip the outliers

A max 2.8× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Jul 19, 2024

What age actually contains in clinical-trial

900 rows, 69 distinct values, and the one fact about age that changes how you query it.

quality practice Data quality

Jul 18, 2024

release_year by genre: a 0% spread

Real group means for one column across 8 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jul 17, 2024

Charting line_no by country

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Jul 16, 2024

Reader question — Designing an experiment that can succeed

Most experiments fail before launch, because they were never large enough to detect the effect anyone hoped for. The fix costs ten minutes and happens before you write code.

practice Experimentation

Jul 15, 2024

clinical-trial.adverse_event: 4 values, 97.8% in the top three

4 values with 64.8% concentrated in one of them, and what that does to every chart downstream.

visualization practice Visualization

Jul 14, 2024

Reading posting_id before trusting it in data-job-postings

3,500 rows, all distinct, and the one fact about posting_id that changes how you query it.

quality practice Data quality

Jul 13, 2024

Date spine, in short

Join to a generated calendar so days with no activity appear as zero instead of vanishing.

sql Analytics engineering

Jul 12, 2024

Is ticket_id + priority the grain of support-tickets?

4,000 distinct combinations against 4,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Jul 11, 2024

Five minutes with baseline_score in clinical-trial

900 rows, 384 distinct values, and the one fact about baseline_score that changes how you query it.

quality practice Data quality

Jul 10, 2024

The overall average hides 5 different numbers

Real group means for one column across 5 segments, and what the pooled average conceals.

statistics practice Analytics practice

Jul 9, 2024

Before you clip the outliers

A max 1.6× the 99th percentile, and the three different things that could be.

statistics quality Statistics

Jul 8, 2024

What queue actually contains in support-tickets

4,000 rows, 5 distinct values, and the one fact about queue that changes how you query it.

quality practice Data quality

Jul 7, 2024

A strong correlation, and what it is not

A measured correlation of 0.98 between two columns, what its square says, and the diagnostic that costs one line.

statistics Statistics

Jul 6, 2024

Charting duration_min by borough

Two facts about the columns decide the chart. Cardinality and skew, in that order.

visualization Visualization

Jul 5, 2024

Five minutes with site in clinical-trial

900 rows, 6 distinct values, and the one fact about site that changes how you query it.

quality practice Data quality

Jul 4, 2024

Is ticket_id + resolved_at the grain of support-tickets?

4,000 distinct combinations against 4,000 rows, and what that answers about the table's key.

sql quality warehousing Analytics engineering

Jul 3, 2024

Cutting a lesson down — Batch, streaming, and the honest difference

Streaming is not "batch but faster". It is a different consistency model with different failure modes, and most teams that adopt it did not need to.

practice Data engineering