Batch, streaming, and the honest difference

The question "should this be streaming?" gets answered by fashion far more often than by requirements. Here is the version that holds up.

The actual distinction#

Batch processes a bounded set of data: everything that arrived between two points in time. The set is complete before processing starts. You know how many rows there are.

Streaming processes an unbounded set: records arrive continuously and processing never finishes. You never know whether more data for a given period is still coming.

That second sentence is the whole difficulty. Not throughput, not latency — completeness. In batch, "all of Tuesday" is a thing you can wait for. In streaming, "all of Tuesday" is a bet you place with a watermark and an allowance for late arrivals.

The honest cost comparison#

None of that says never stream. It says streaming's cost is paid in operational complexity, permanently, and that cost should buy a decision that genuinely needs the latency.

Micro-batch: where most teams should actually live#

Between the two sits micro-batch: run the batch job every one, five, or fifteen minutes. You keep batch's reasoning model — bounded sets, easy reruns, trivial debugging — and get most of the freshness.

An enormous share of "we need streaming" requirements are met by a five-minute micro-batch, at roughly a tenth of the operational cost. Ask what happens if the data is four minutes old. Usually: nothing.

Event time versus processing time#

This is the concept that makes streaming hard, and it also bites batch pipelines that nobody warned.

• Event time: when the thing happened. A user tapped "buy" at 23:58.
• Processing time: when your system saw it. Their phone was offline; it arrived at 00:07.

That purchase belongs to yesterday's revenue by event time and today's by processing time. Both are defensible. Only one is what your finance team means. If you never decide explicitly, you have decided implicitly — and it is almost always processing time, which is almost always wrong.

-- Processing time: fast, stable, and wrong for anything a human reconciles.
select date_trunc('day', _loaded_at) as day, sum(revenue) ...

-- Event time: correct, but yesterday's total can still change tomorrow.
select date_trunc('day', ordered_at) as day, sum(revenue) ...

The second query has a property people find unacceptable until it is explained: the past changes. Yesterday's revenue goes up slightly for a couple of days as late events land. That is not a bug; it is the data telling the truth. The fix is to say so — publish the restatement window ("figures firm after 72 hours") rather than pretend it does not happen.

Watermarks, in one paragraph#

A watermark is a claim: "I believe I have now seen all events with an event time earlier than T." It lets a streaming system close a window and emit a result. It is a heuristic, so it is sometimes wrong, so systems also define allowed lateness — how long after closing a window they will still accept and reprocess a straggler. Tuning those two knobs is most of the work in real streaming pipelines, and getting them wrong shows up as a metric that quietly under-counts at the edges.

Choosing, in practice#

Reach for batch by default. It is easier to reason about, cheaper, and simpler to fix. Reach for micro-batch when someone wants fresher numbers. Reach for streaming when a machine — not a person — consumes the output and acts within seconds: fraud decisions, real-time personalisation, alerting on a live system, anything with a hard SLA measured in seconds.

And when you do stream, keep a batch job that recomputes the same numbers daily from the raw log. It is your correctness backstop, and the day it disagrees with the stream you will be extremely glad it exists.