Field note · October 14, 2026
Do I need to learn Spark?
Probably not first, and possibly not at all.
A question that came in more than once, so the answer goes here.
Do I need to learn Spark?
Probably not first, and possibly not at all.
Most data that people reach for Spark to process fits on one machine. A modern laptop handles tens of millions of rows in DuckDB or Polars comfortably, and it does it without a cluster, a scheduler, or a bill.
Learn it when you have a workload that genuinely does not fit — and know the number, so "does not fit" is a measurement rather than a vibe. Learning it before that means learning distributed-systems debugging in order to solve a problem you do not have.
What transfers regardless of engine: partitioning, file layout, predicate pushdown, and why a shuffle is expensive. Those show up in Why your query costs what it costs and are worth more than any specific API.
The counter-argument, which is fair: plenty of job postings list it, and you cannot interview for a role whose keyword filter you did not pass. If that is the situation, learn enough to talk about the execution model honestly — what a shuffle costs, why a broadcast join helps, what causes skew — rather than memorising DataFrame methods. The first set survives a follow-up question and the second does not.