Is my pipeline actually idempotent?
There is one test. Run it twice on the same input and diff the output. If it differs, it is not.
Tag
pipelines
17 pieces tagged “pipelines”, newest first.
Merge with a recency guard, in short
Upsert on a natural key, and refuse to let an out-of-order replay overwrite good data with stale data.
Idempotent window replace, in short
Every write replaces a named slice rather than adding to a pile, so rerunning a job is safe and backfilling is routine.
Pattern: Deterministic surrogate key
Derive the key from the content instead of an auto-increment, so reprocessing the same row produces the same key.
Idempotent window replace, in short
Every write replaces a named slice rather than adding to a pile, so rerunning a job is safe and backfilling is routine.
Contract check at the boundary, in short
Validate the shape of incoming data where it enters your system — fail loudly on structure, warn on distribution.
Lookback window, in short
Reprocess the last N days every night so late-arriving events are picked up instead of landing in a closed window.
Deterministic surrogate key, in short
Derive the key from the content instead of an auto-increment, so reprocessing the same row produces the same key.
Pattern: Contract check at the boundary
Validate the shape of incoming data where it enters your system — fail loudly on structure, warn on distribution.
Pattern: Lookback window
Reprocess the last N days every night so late-arriving events are picked up instead of landing in a closed window.
Lookback window, in short
Reprocess the last N days every night so late-arriving events are picked up instead of landing in a closed window.
Contract check at the boundary, in short
Validate the shape of incoming data where it enters your system — fail loudly on structure, warn on distribution.
We were wrong about streaming
Two years of Kafka for a use case that a five-minute batch would have served.
Pattern: Idempotent window replace
Every write replaces a named slice rather than adding to a pile, so rerunning a job is safe and backfilling is routine.
Merge with a recency guard, in short
Upsert on a natural key, and refuse to let an out-of-order replay overwrite good data with stale data.
Yesterday keeps changing and that is correct
Late-arriving events, a seven-day lookback, and how to tell finance about it without a crisis.
Schema drift, on a Friday, obviously
A vendor renamed a field. We found out from a stakeholder. Here is the boundary check we should have had.