Archive
2024
366 field notes published in 2024.
December 31
- An exercise on ab-test-checkout Data quality
- Course note — One metric, one definition Analytics engineering
- Five minutes with flight_date in flight-delays Data quality
- Difference-in-differences, in short Experimentation
- The category that eats the chart Visualization
- Five minutes with adverse_event in clinical-trial Data quality
- Finding the gaps in ride-hail-trips Analytics engineering
- Counting rows is not testing grain Analytics engineering
- The top of solar_mw is 1.2× the 99th percentile Statistics
- The overall average hides 2 different numbers Analytics practice
- Five minutes with o3_ppb in city-air-quality Data quality
- r = 0.70 between fare_usd and tip_usd Statistics
- Charting dep_delay_min by origin Visualization
- Reading bmi before trusting it in clinical-trial Data quality
- Year two Analytics practice
- Cutting a lesson down — Facts, dimensions, and picking a grain Analytics engineering
- Forward-chaining split, in short Machine learning
- Reading category before trusting it in retail-orders Data quality
- The overall average hides 5 different numbers Analytics practice
- movie-ratings.genre: 8 values, 54.8% in the top three Visualization
- Five minutes with churned_at in saas-subscriptions Data quality
- A running total that does not lie about ties Analytics engineering
- A strong correlation, and what it is not Statistics
- Reading line_no before trusting it in retail-orders Data quality
- Counting rows is not testing grain Analytics engineering
- The top of temp_c is 1.4× the 99th percentile Statistics
- Reading surge_multiplier before trusting it in ride-hail-trips Data quality
- Charting gdp_per_capita_usd by region Visualization
- Cutting a lesson down — Features, and explaining what the model did Machine learning
- Five minutes with sex in clinical-trial Data quality
- Median beside the mean, in short Statistics
November 30
- The category that eats the chart Visualization
- Reading distance_km before trusting it in ride-hail-trips Data quality
- Comparing a period to the one before it Analytics engineering
- Before you clip the outliers Statistics
- Three dashboards, three revenue numbers, ninety minutes Analytics engineering
- What user_id actually contains in ab-test-checkout Data quality
- Counting rows is not testing grain Analytics engineering
- mrr_usd by plan: a 13,612% spread Analytics practice
- Five minutes with humidity_pct in sensor-telemetry Data quality
- A strong correlation, and what it is not Statistics
- Charting csat by queue Visualization
- Cutting a lesson down — Writing SQL people can read, and reading the plan Analytics engineering
- Reading humidity_pct before trusting it in sensor-telemetry Data quality
- Pattern: Difference-in-differences Experimentation
- flight-delays.carrier: 6 values, 63.6% in the top three Visualization
- Reading country before trusting it in retail-orders Data quality
- Sessions out of raw events in ride-hail-trips Analytics engineering
- Counting rows is not testing grain Analytics engineering
- Five minutes with dropped_out in clinical-trial Data quality
- The top of wind_mw is 1.6× the 99th percentile Statistics
- The overall average hides 3 different numbers Analytics practice
- What sessions actually contains in ab-test-checkout Data quality
- r = 0.00 between no2_ppb and o3_ppb Statistics
- Charting bmi by site Visualization
- What resolved_at actually contains in support-tickets Data quality
- data-job-postings.seniority: 5 values, 83.4% in the top three Visualization
- Cutting a lesson down — Choosing the form before the colours Visualization
- Five minutes with carrier in flight-delays Data quality
- As-of feature computation, in short Machine learning
- Sessions out of raw events in data-job-postings Analytics engineering
October 31
- Five minutes with queue in support-tickets Data quality
- Counting rows is not testing grain Analytics engineering
- Two more tools, and why the palette is not our taste Visualization
- The top of revenue_usd is 3.0× the 99th percentile Statistics
- What tip_usd actually contains in ride-hail-trips Data quality
- no2_ppb by city: a 129% spread Analytics practice
- A weak correlation, and what it is not Statistics
- Five minutes with solar_mw in grid-energy-load Data quality
- Charting seats by region Visualization
- The category that eats the chart Visualization
- What posting_id actually contains in data-job-postings Data quality
- Reader question — Orchestration, and what "it runs every night" costs Data platform
- Gaps and islands, in short Analytics engineering
- What revenue_usd actually contains in ab-test-checkout Data quality
- Finding the gaps in retail-orders Analytics engineering
- Counting rows is not testing grain Analytics engineering
- What is_holiday actually contains in grid-energy-load Data quality
- r = 0.19 between year and population Statistics
- Before you clip the outliers Statistics
- Reading pm10 before trusting it in city-air-quality Data quality
- The overall average hides 3 different numbers Analytics practice
- Charting population by region Visualization
- Counting rows is not testing grain Analytics engineering
- Reading pm25 before trusting it in city-air-quality Data quality
- Cutting a lesson down — Why your query costs what it costs Data platform
- Pattern: Lookback window Data engineering
- The top of salary_min_usd is 1.2× the 99th percentile Statistics
- pm10 by city: a 174% spread Analytics practice
- Reading is_holiday before trusting it in grid-energy-load Data quality
- sensor-telemetry.site: 3 values, 100.0% in the top three Visualization
- Finding the gaps in flight-delays Analytics engineering
September 30
- Five minutes with company_size in data-job-postings Data quality
- A moderate correlation, and what it is not Statistics
- Charting duration_min by payment_type Visualization
- The category that eats the chart Visualization
- What city actually contains in data-job-postings Data quality
- Course note — Idempotency, backfills, and yesterday arriving twice Data engineering
- We rewrote our interview loop and the hires got better Analytics practice
- Difference-in-differences, in short Experimentation
- Five minutes with region in world-indicators Data quality
- Is reading_at + site the grain of sensor-telemetry? Analytics engineering
- The top of price_eur_mwh is 1.2× the 99th percentile Statistics
- Five minutes with fare_usd in ride-hail-trips Data quality
- The overall average hides 4 different numbers Analytics practice
- A strong correlation, and what it is not Statistics
- What wind_kph actually contains in city-air-quality Data quality
- Charting salary_min_usd by seniority Visualization
- Reader question — Choosing the form before the colours Visualization
- Five minutes with plan in saas-subscriptions Data quality
- clinical-trial.arm: 2 values, 100.0% in the top three Visualization
- Anti-join, in short Analytics engineering
- Five minutes with rated_at in movie-ratings Data quality
- Is order_id + category the grain of retail-orders? Analytics engineering
- The overall average hides 6 different numbers Analytics practice
- Five minutes with remote in data-job-postings Data quality
- Before you clip the outliers Statistics
- A essentially none correlation, and what it is not Statistics
- Five minutes with bmi in clinical-trial Data quality
- Charting seats_now by region Visualization
- Course note — Causal claims when you cannot randomise Experimentation
- Reading load_mw before trusting it in grid-energy-load Data quality
August 31
- The top of population is 1.1× the 99th percentile Statistics
- The category that eats the chart Visualization
- Reading pickup_at before trusting it in ride-hail-trips Data quality
- The grain uniqueness test, in short Data quality
- A backfill that cost $400 in ninety seconds Data platform
- The overall average hides 5 different numbers Analytics practice
- What seats actually contains in saas-subscriptions Data quality
- Is user_id + device the grain of ab-test-checkout? Analytics engineering
- r = 0.53 between fare_usd and surge_multiplier Statistics
- Five minutes with internet_pct in world-indicators Data quality
- Charting dep_delay_min by carrier Visualization
- Course note — Monitoring data, not just jobs Data platform
- Sessionisation, in short Analytics engineering
- The top of co2_tonnes_per_capita is 1.3× the 99th percentile Statistics
- What mrr_usd actually contains in saas-subscriptions Data quality
- data-job-postings.company_size: 4 values, 81.1% in the top three Visualization
- Counting rows is not testing grain Analytics engineering
- Reading age before trusting it in clinical-trial Data quality
- gdp_per_capita_usd by country: a 3,670% spread Analytics practice
- A moderate correlation, and what it is not Statistics
- Reading flight_date before trusting it in flight-delays Data quality
- Charting mrr_usd by region Visualization
- Reader question — Schema drift, and the contract that prevents it Data engineering
- Reading payment_type before trusting it in ride-hail-trips Data quality
- Slowly changing dimension, type 2, in short Analytics engineering
- Is posting_id + city the grain of data-job-postings? Analytics engineering
- Reading region before trusting it in world-indicators Data quality
- seats_now by plan: a 25,267% spread Analytics practice
- The category that eats the chart Visualization
- What remote actually contains in data-job-postings Data quality
- The top of sessions is 2.8× the 99th percentile Statistics
July 31
- saas-subscriptions.industry: 8 values, 39.8% in the top three Visualization
- Writing a SQL engine that fits in a browser tab Visualization
- Reading units before trusting it in retail-orders Data quality
- A essentially none correlation, and what it is not Statistics
- Charting age by site Visualization
- Five minutes with seats in saas-subscriptions Data quality
- Reader question — A working setup that will not embarrass you Data engineering
- Lookback window, in short Data engineering
- r = 0.07 between age and bmi Statistics
- Reading plan before trusting it in saas-subscriptions Data quality
- Is line_no + channel the grain of retail-orders? Analytics engineering
- Before you clip the outliers Statistics
- What age actually contains in clinical-trial Data quality
- release_year by genre: a 0% spread Analytics practice
- Charting line_no by country Visualization
- Reader question — Designing an experiment that can succeed Experimentation
- clinical-trial.adverse_event: 4 values, 97.8% in the top three Visualization
- Reading posting_id before trusting it in data-job-postings Data quality
- Date spine, in short Analytics engineering
- Is ticket_id + priority the grain of support-tickets? Analytics engineering
- Five minutes with baseline_score in clinical-trial Data quality
- The overall average hides 5 different numbers Analytics practice
- Before you clip the outliers Statistics
- What queue actually contains in support-tickets Data quality
- A strong correlation, and what it is not Statistics
- Charting duration_min by borough Visualization
- Five minutes with site in clinical-trial Data quality
- Is ticket_id + resolved_at the grain of support-tickets? Analytics engineering
- Cutting a lesson down — Batch, streaming, and the honest difference Data engineering
- Fan-out join, in short Analytics engineering
- clinical-trial.site: 6 values, 52.7% in the top three Visualization
June 30
- Before you clip the outliers Statistics
- Five minutes with release_year in movie-ratings Data quality
- Charting distance_km by payment_type Visualization
- The overall average hides 4 different numbers Analytics practice
- What reading_date actually contains in city-air-quality Data quality
- The missing values were the finding Statistics
- r = 0.77 between gdp_per_capita_usd and life_expectancy Statistics
- fare_usd by borough: a 229% spread Analytics practice
- What discount_pct actually contains in retail-orders Data quality
- Reader question — The five tests that cover most questions Experimentation
- Contract check at the boundary, in short Data quality
- What passengers actually contains in ride-hail-trips Data quality
- saas-subscriptions.channel: 6 values, 60.9% in the top three Visualization
- Is movie_id + rated_at the grain of movie-ratings? Analytics engineering
- Before you clip the outliers Statistics
- A essentially none correlation, and what it is not Statistics
- Five minutes with mrr_usd in saas-subscriptions Data quality
- Charting humidity_pct by site Visualization
- Reader question — Features, and explaining what the model did Machine learning
- Five minutes with converted in ab-test-checkout Data quality
- Baseline before model, in short Machine learning
- Before you clip the outliers Statistics
- What release_year actually contains in movie-ratings Data quality
- data-job-postings.remote: 3 values, 100.0% in the top three Visualization
- Counting rows is not testing grain Analytics engineering
- Reading week12_score before trusting it in clinical-trial Data quality
- sessions by variant: a 2% spread Analytics practice
- r = 0.00 between flight_no and sched_dep_hour Statistics
- What country actually contains in retail-orders Data quality
- Charting salary_max_usd by city Visualization
May 31
- Reader question — Slowly changing dimensions, and the "as of when" problem Analytics engineering
- Five minutes with tip_usd in ride-hail-trips Data quality
- Pattern: Bootstrap any statistic Statistics
- The category that eats the chart Visualization
- Reading population before trusting it in world-indicators Data quality
- Counting rows is not testing grain Analytics engineering
- The top of week12_score is 1.2× the 99th percentile Statistics
- What channel actually contains in saas-subscriptions Data quality
- discount_pct by channel: a 280% spread Analytics practice
- A moderate correlation, and what it is not Statistics
- We were wrong about streaming Data engineering
- What price_eur_mwh actually contains in grid-energy-load Data quality
- Reader question — Colour is an encoding, not decoration Visualization
- Conditional pivot, in short Analytics engineering
- Reading dep_delay_min before trusting it in flight-delays Data quality
- support-tickets.queue: 5 values, 72.2% in the top three Visualization
- Counting rows is not testing grain Analytics engineering
- Before you clip the outliers Statistics
- What sched_dep_hour actually contains in flight-delays Data quality
- A essentially none correlation, and what it is not Statistics
- The top of fare_usd is 2.4× the 99th percentile Statistics
- Five minutes with genre in movie-ratings Data quality
- The overall average hides 5 different numbers Analytics practice
- Course note — Facts, dimensions, and picking a grain Analytics engineering
- r = 0.73 between distance_km and fare_usd Statistics
- Reading genre before trusting it in movie-ratings Data quality
- Counting rows is not testing grain Analytics engineering
- support-tickets.priority: 4 values, 92.6% in the top three Visualization
- Reading company_size before trusting it in data-job-postings Data quality
- Is trip_id + payment_type the grain of ride-hail-trips? Analytics engineering
- The overall average hides 3 different numbers Analytics practice
April 30
- Reading region before trusting it in saas-subscriptions Data quality
- retail-orders.channel: 4 values, 83.6% in the top three Visualization
- The top of distance_km is 1.7× the 99th percentile Statistics
- Five minutes with country in data-job-postings Data quality
- Pattern: The grain uniqueness test Data quality
- The overall average hides 6 different numbers Analytics practice
- Reading fare_usd before trusting it in ride-hail-trips Data quality
- The `else` branch that ate 4% of revenue Data quality
- Cutting a lesson down — Monitoring data, not just jobs Data platform
- Pattern: Idempotent window replace Data engineering
- retail-orders.country: 8 values, 61.5% in the top three Visualization
- What surge_multiplier actually contains in ride-hail-trips Data quality
- r = 0.03 between humidity_pct and vibration_mm_s Statistics
- Cutting a lesson down — Correlation, confounding, and Simpson's paradox Statistics
- What carrier actually contains in flight-delays Data quality
- Is order_id + country the grain of retail-orders? Analytics engineering
- The left join that became an inner join, in short Analytics engineering
- Before you clip the outliers Statistics
- flight_no by origin: a 8% spread Analytics practice
- What revenue_usd actually contains in retail-orders Data quality
- r = 0.00 between temp_c and rpm Statistics
- Course note — The first hour with an unfamiliar dataset Data quality
- Pattern: The left join that became an inner join Analytics engineering
- The category that eats the chart Visualization
- Is movie_id + user_id the grain of movie-ratings? Analytics engineering
- Reading ticket_id before trusting it in support-tickets Data quality
- The top of salary_max_usd is 1.2× the 99th percentile Statistics
- The overall average hides 2 different numbers Analytics practice
- Five minutes with temp_c in sensor-telemetry Data quality
- r = 0.56 between duration_min and tip_usd Statistics
March 31
- Course note — Choosing the form before the colours Visualization
- Baseline before model, in short Machine learning
- Is device_id + site the grain of sensor-telemetry? Analytics engineering
- Before you clip the outliers Statistics
- Reading is_returning before trusting it in ab-test-checkout Data quality
- first_response_min by priority: a 1,095% spread Analytics practice
- r = 0.98 between seats and seats_now Statistics
- What temp_c actually contains in grid-energy-load Data quality
- Reader question — Correlation, confounding, and Simpson's paradox Statistics
- Bootstrap any statistic, in short Statistics
- What ticket_id actually contains in support-tickets Data quality
- bmi by site: a 3% spread Analytics practice
- The chart builder is live, and it refuses to do things Visualization
- Is user_id + rated_at the grain of movie-ratings? Analytics engineering
- Five minutes with reading_at in sensor-telemetry Data quality
- The top of seats_now is 1.5× the 99th percentile Statistics
- r = 0.01 between age and week12_score Statistics
- What pickup_at actually contains in ride-hail-trips Data quality
- Cutting a lesson down — Slowly changing dimensions, and the "as of when" problem Analytics engineering
- Merge with a recency guard, in short Data engineering
- What genre actually contains in movie-ratings Data quality
- vibration_mm_s by status: a 191% spread Analytics practice
- Is flight_no + origin the grain of flight-delays? Analytics engineering
- What company_size actually contains in data-job-postings Data quality
- Before you clip the outliers Statistics
- r = -0.02 between humidity_pct and rpm Statistics
- Reading year before trusting it in world-indicators Data quality
- Course note — Uncertainty, sampling, and how much to trust a number Statistics
- Is line_no + order_id the grain of retail-orders? Analytics engineering
- What load_mw actually contains in grid-energy-load Data quality
- Sessionisation, in short Analytics engineering
February 29
- Before you clip the outliers Statistics
- Is device_id + reading_at the grain of sensor-telemetry? Analytics engineering
- Five minutes with temp_c in grid-energy-load Data quality
- The overall average hides 3 different numbers Analytics practice
- r = 0.09 between duration_min and surge_multiplier Statistics
- What vibration_mm_s actually contains in sensor-telemetry Data quality
- Reader question — Facts, dimensions, and picking a grain Analytics engineering
- Pattern: Sessionisation Analytics engineering
- Before you clip the outliers Statistics
- We shipped a Simpson's paradox to the exec team Statistics
- Five minutes with vibration_mm_s in sensor-telemetry Data quality
- seats_now by region: a 20% spread Analytics practice
- A essentially none correlation, and what it is not Statistics
- What variant actually contains in ab-test-checkout Data quality
- Course note — Describing a column without lying Statistics
- The grain uniqueness test, in short Data quality
- Five minutes with no2_ppb in city-air-quality Data quality
- Counting rows is not testing grain Analytics engineering
- Before you clip the outliers Statistics
- Five minutes with prior_therapy in clinical-trial Data quality
- rating by genre: a 16% spread Analytics practice
- A essentially none correlation, and what it is not Statistics
- What churned_at actually contains in saas-subscriptions Data quality
- Cutting a lesson down — Turning a request into a question Analytics practice
- Counting rows is not testing grain Analytics engineering
- The top of gdp_per_capita_usd is 1.4× the 99th percentile Statistics
- What seats_now actually contains in saas-subscriptions Data quality
- A proportion with an interval, in short Statistics
- The overall average hides 3 different numbers Analytics practice
January 31
- Five minutes with posting_id in data-job-postings Data quality
- A weak correlation, and what it is not Statistics
- Reader question — Communicating a result to someone who will not read your notebook Visualization
- Five minutes with device in ab-test-checkout Data quality
- Counting rows is not testing grain Analytics engineering
- sched_dep_hour by dest: a 8% spread Analytics practice
- What temp_c actually contains in sensor-telemetry Data quality
- r = -0.25 between temp_c and wind_mw Statistics
- Is account_id + industry the grain of saas-subscriptions? Analytics engineering
- What country actually contains in data-job-postings Data quality
- r = 0.11 between sched_dep_hour and dep_delay_min Statistics
- Cutting a lesson down — One metric, one definition Analytics engineering
- Five minutes with trip_id in ride-hail-trips Data quality
- seats by industry: a 53% spread Analytics practice
- Is flight_no + dest the grain of flight-delays? Analytics engineering
- A timezone bug that cost us three days Data engineering
- Reading solar_mw before trusting it in grid-energy-load Data quality
- Pattern: Anti-join Analytics engineering
- Cutting a lesson down — The anatomy of a silent failure Data quality
- The overall average hides 14 different numbers Analytics practice
- Reading hour_at before trusting it in grid-energy-load Data quality
- A strong correlation, and what it is not Statistics
- Cutting a lesson down — A working setup that will not embarrass you Data engineering
- What dropped_out actually contains in clinical-trial Data quality
- Counting rows is not testing grain Analytics engineering
- mrr_usd by region: a 21% spread Analytics practice
- r = 0.08 between population and life_expectancy Statistics
- Reading prior_therapy before trusting it in clinical-trial Data quality
- Course note — Reading a result without fooling yourself Experimentation
- Counting rows is not testing grain Analytics engineering
- The overall average hides 6 different numbers Analytics practice