Schema drift, on a Friday, obviously

A payments vendor renamed txn_id to transaction_id in a minor API version bump. Their changelog described it as non-breaking, which from their side it was.

Our ingestion did select * into a raw table, so the load succeeded. The staging model referenced txn_id, which was now missing, so it produced nulls. Nulls flowed into a join key. The join silently dropped 100% of the day's payments.

Revenue for that day showed as zero on Saturday morning. We noticed on Monday, from a message that started "is the dashboard broken or did we have a very bad weekend".

The check that would have caught it at 04:00 Saturday is about fifteen lines:

def validate(df, contract):
    problems = []
    for spec in contract:
        if spec.name not in df.columns:
            problems.append(f"missing column: {spec.name}")
    extra = set(df.columns) - {s.name for s in contract}
    if extra:
        print(f"note: new upstream columns ignored: {sorted(extra)}")
    if problems:
        raise ValueError("Contract violation:\n  " + "\n  ".join(problems))

Note the asymmetry, which took us a while to get right. A missing column is fatal. A new column is logged and ignored. Additive change should never wake anyone up; a removal always should.

The other change: our ingestion no longer does select * into raw. It names every column it depends on, which converts a silent semantic failure into a loud structural one. That trade is almost always worth making.