The join that doubled revenue for six weeks

A colleague joined an orders fact table to a customer attributes table. The attributes table had, for about 8% of customers, two rows — one from a legacy migration and one current. Nobody knew.

Revenue for those customers doubled. Total revenue went up about 4%, which is inside the range where "we had a good month" is a plausible explanation. It stayed wrong for six weeks.

The controls we had, and why each missed it:

Row count monitoring. Watched the source tables, not the joined model. Both sources were fine.

Null checks. Nothing was null. Every value was present and every one appeared twice.

A dashboard someone looks at daily. They looked at it daily. It went up. Nobody investigates a number going up.

Reconciliation against billing. We did not have this. It is what eventually found it, when someone did it manually for an unrelated reason.

The fix is embarrassingly small. Every model now carries a uniqueness test on its declared grain:

models:
  - name: fct_order_lines
    description: One row per order line.
    columns:
      - name: order_line_key
        tests: [unique, not_null]

That is it. One test per model, non-negotiable, no exceptions. It directly encodes the grain sentence, and it would have failed on the first run after the bad join.

The lesson I actually took away is not about testing. It is that a number moving in the direction you want gets no scrutiny, and that asymmetry is where the expensive bugs live.