Is my pipeline actually idempotent?

A question that came in more than once, so the answer goes here.

Is my pipeline actually idempotent?

There is one test. Run it twice on the same input and diff the output. If it differs, it is not.

Most pipelines fail this and nobody finds out until a backfill. The usual causes: an append where an upsert belonged, a key derived from something that changes between runs, or a "load everything since last run" watermark that moves before the load is confirmed.

# the whole test
run(partition="2024-03-01")
first = snapshot(table)
run(partition="2024-03-01")
assert snapshot(table) == first

The reason it matters is not elegance. It is that a pipeline that cannot be safely rerun cannot be safely fixed — every recovery becomes a manual operation performed by someone tired, at the worst possible time, under the most pressure.

The property to aim for is that re-running any partition produces the same result regardless of how many times it has run before, and regardless of what order the partitions ran in. That second half is the one people miss: a pipeline can be idempotent per-run and still be order-dependent, which breaks the moment you backfill six months in parallel.

Idempotency and backfills covers the patterns.