The anatomy of a silent failure

Loud failures are fine. A job crashes, someone gets paged, it gets fixed, and the total damage is a few hours of stale data. The incidents that hurt are the ones where everything reports success.

Here is the shape of one. The details are composite; the pattern is not.

Week zero: a reasonable change upstream#

The payments team adds a new payment method. Their payment_type enum gains a value: bnpl. They ship it on a Tuesday. They do not tell the data team, because they do not know the data team reads that column — and honestly, why would they.

Week zero, still: the silent swallow#

Downstream, a revenue model has this:

case
    when payment_type in ('card', 'wallet') then revenue_usd
    when payment_type = 'cash'              then revenue_usd * 0.98   -- cash handling fee
    else 0
end as net_revenue

bnpl hits the else. Every buy-now-pay-later order now contributes exactly zero to net revenue.

Nothing fails. The pipeline is green. Row counts are normal — the rows are all there, they just contain a zero. Uniqueness passes, not-null passes, referential integrity passes. Freshness is fine. The volume test compares row counts, and the row count is correct.

Weeks one to five: the slow divergence#

BNPL starts at 0.4% of orders and grows. Net revenue drifts below where it should be, by an amount that starts inside normal weekly noise and stays there for a month. The weekly number is down 1.2%, then 2.1%, then 3.4%. Each week is individually unremarkable. Somebody notes that growth "feels soft".

Week six: discovery, by accident#

A finance analyst reconciling against the billing system finds a 4.8% gap. Not through a test. Through a human doing a manual comparison for an unrelated reason.

The post-mortem#

Six things had to be true for this to happen, and each one is a control that was missing.

1. No accepted-values test on payment_type. The cheapest possible fix. One test, five lines of YAML, would have caught this on day one.

2. A silent else branch. The case statement chose to be forgiving. In a revenue calculation, forgiving means wrong. It should have been:

case
    when payment_type in ('card', 'wallet') then revenue_usd
    when payment_type = 'cash'              then revenue_usd * 0.98
    -- New payment types must be handled explicitly. Fail rather than guess.
    else error('unhandled payment_type: ' || payment_type)
end

Engines that lack an error() function can do the same with a division by zero or a separate test asserting the else branch is empty. The point is that the default is stop, not zero.

3. No distribution monitoring. The share of rows landing in the else branch went from 0% to 4%. That is a trivially detectable change if anyone is measuring category proportions rather than only row counts.

4. No reconciliation. The billing comparison that eventually found it was ad hoc. As a scheduled weekly check with a 1% tolerance, it would have fired in week two.

5. No contract with the payments team. They had no way to know the change mattered. Not their fault — nobody had ever told them, and nothing in their CI mentioned a downstream consumer.

6. "Feels soft" was never escalated. Someone's intuition was correct and had nowhere to go. Worth building a norm where "this number looks wrong to me" is a legitimate ticket rather than a vibe.

The other silent failures, briefly#

Once you know the pattern, you see it everywhere.

A join that started fanning out when a dimension gained a second row per key. Row counts go up, and everyone assumes growth.

A timezone change. A source starts sending UTC instead of local time. Daily totals are now shifted by a few hours, so each day borrows from its neighbour. Weekly totals look perfect. Daily ones are all slightly wrong.

A units change. A vendor switches from dollars to cents. coalesce-and-carry-on means revenue is suddenly 100× — that one gets caught fast. The nastier version is switching from cents to dollars, making everything 1% of its former size in one currency out of nine.

A filter that stopped matching. where region = 'EMEA' after the source renames it to emea. Returns zero rows. If the dashboard shows a zero rather than an error, and it is one panel of twelve, nobody notices for a month.

Deduplication with a non-deterministic tiebreak. row_number() with no unique ordering picks a different row on each run. Numbers change slightly with every rebuild and nobody can reproduce anything.

The defence, condensed#

• Test accepted values on every categorical column that appears in a case.
• Make defaults loud. else should fail, not zero.
• Monitor the distribution, not just the count.
• Reconcile the handful of numbers that matter against an independent system, on a schedule.
• Treat "this looks wrong to me" as a valid bug report.