A backfill that cost $400 in ninety seconds

We fixed a bug in a revenue model and needed to reprocess history. Four hundred days. The backfill script fired them all off at once, because nothing stopped it.

Each run scanned about 80GB. Four hundred runs is 32TB. On a per-byte-priced warehouse that is roughly $400 in the time it takes to make coffee, and it saturated the warehouse for everyone else while doing it.

The fix is boring and I would put it in any backfill script:

# Bounded concurrency. Four at a time is plenty and will not page anyone.
printf '%s\n' "${DATES[@]}" | xargs -P 4 -I{} ./run.sh {}

Three things we added:

A dry-run mode that reports the number of partitions and the estimated bytes without executing. Most warehouses expose a dry-run API; it costs nothing and it would have shown 32TB before we spent it.

Bounded concurrency by default, opt-in to more.

A byte limit on scheduled jobs. A job that suddenly scans 40× more than usual should fail rather than succeed expensively. This has since caught two real regressions.

The broader thing: backfills should be routine, not heroic. If a backfill feels dangerous, your pipeline is not idempotent and that is the actual problem. But "routine" also means knowing what it costs before you start, and we did not.