We were wrong about streaming

We built a streaming pipeline because the requirement said "real-time inventory". Kafka, a stream processor, watermarks, allowed lateness, the whole apparatus. It took a quarter and it worked.

Eighteen months later someone asked what consumed it. Answer: a dashboard that operations staff check about four times a day, and an alert that fires when stock drops below a threshold.

The alert needs latency of minutes. The dashboard needs latency of "before I look at it".

A five-minute micro-batch would have met both requirements at maybe a tenth of the operational cost. We paid for two years of: an on-call rotation, watermark tuning, a topic retention decision nobody wanted to own, and debugging that meant reading logs rather than querying a table.

The mistake was accepting "real-time" as a requirement instead of asking the question behind it: what decision is made on this data, and how quickly does that decision go stale?

Nobody had asked. "Real-time" sounds like a specification and is actually a vibe.

What I ask now, before any freshness requirement is accepted:

• Who or what consumes this?
• If it were four minutes old, what would go wrong?
• Is the consumer a human or a machine? Humans essentially never need seconds.

The honest ordering is: batch by default, micro-batch when someone wants fresher, streaming only when a machine acts within seconds. Fraud blocks, live personalisation, alerting on a system. Not dashboards.

We have since replaced it with a five-minute batch. Nobody noticed, which is the whole point.