DuckDB replaced a Spark cluster for one of our jobs

We had a nightly job on Spark. Three executors, about 40 minutes wall clock including cluster spin-up, processing roughly 4GB of Parquet into an aggregate.

Someone rewrote it as a DuckDB query on a single machine as a joke. It ran in 90 seconds.

The data was never big. It was 4GB. We were running a distributed system because the job had been written when we thought we would have a hundred times more data, and we never revisited the decision when we did not.

What we actually paid for that:

• Cluster cost, nightly, for three years.
• Startup overhead longer than the useful work.
• Debugging that meant reading stage graphs instead of a stack trace.
• One person who understood the tuning, and a bus factor of one.

The threshold I use now: if a day's partition fits in the memory of one large cloud instance, you do not need a cluster. That is a few hundred gigabytes today. It rules out most of what runs on Spark.

Spark is genuinely excellent when the data is genuinely large, or when you need streaming. Both of those are real and neither applied here.

The uncomfortable general version: most infrastructure decisions are made at the moment of maximum ignorance about the workload, and almost nobody revisits them when the facts arrive.