Field note · October 7, 2024

Cutting a lesson down — Why your query costs what it costs

Columnar storage, partitioning, and file layout explain most of the difference between a query that returns in two seconds and the same query taking four minutes and forty dollars.

1 min read ·Data platform ·practice

A note from writing Why your query costs what it costs, which took three passes to get to something short.

Columnar storage, partitioning, and file layout explain most of the difference between a query that returns in two seconds and the same query taking four minutes and forty dollars.

The hard part of writing this was cutting it. The first version covered every case; the useful version covers the case you hit on a Tuesday and names the rest in a sentence. Completeness is a property of reference material, not of teaching material, and confusing the two produces something nobody finishes.

It sits in the Modelling the warehouse module. The exercise is a thinking one rather than a query one — some of this material is about deciding what to compute, and that does not need a keyboard.

Ordering matters here more than in most courses. It follows Slowly changing dimensions, and the "as of when" problem and leads into One metric, one definition, and reading it out of sequence mostly works but costs you the setup.

Free means free, and it also means we can rewrite it whenever it is wrong. No edition, no errata PDF, nothing to repurchase. The whole course is 36 lessons and the fixes land the day we find them.