Schema drift, and the contract that prevents it

Sources change. A field gets renamed, a type widens, an enum gains a value, a nested object becomes an array. None of this is malice — the upstream team is shipping features and has no idea you exist.

The question is not how to prevent drift. It is how you find out: at ingestion, loudly, or three weeks later when someone notices the dashboard has been wrong.

The four kinds of drift, by how nasty they are#

Additive. A new column appears. Harmless if you select explicitly, and a good argument for never using select * in a pipeline.

Renaming. user_id becomes customer_id. Loud and immediate if you select explicitly — the job fails, which is the good outcome.

Type change. An integer becomes a string, or a string becomes nullable. This is the dangerous one, because many systems will silently coerce. A numeric ID that arrives as "00123" and gets cast to 123 will fail to join, quietly, against rows that were already there.

Semantic change. The column keeps its name and type, and changes meaning. status = 'active' used to include trials and now does not. Nothing anywhere will catch this. No schema check, no type system, no test that only looks at structure. The only defences are a relationship with the upstream team and distribution monitoring — watching whether the proportion of active rows jumped overnight.

Validate at the boundary#

Check the shape of data where it enters your system, not where it is consumed. One check at the ingestion boundary protects every downstream model.

from dataclasses import dataclass

@dataclass(frozen=True)
class ColumnSpec:
    name: str
    dtype: str
    nullable: bool = False

ORDERS_CONTRACT = [
    ColumnSpec("order_id", "string"),
    ColumnSpec("ordered_at", "timestamp"),
    ColumnSpec("customer_id", "string"),
    ColumnSpec("revenue_usd", "float"),
    ColumnSpec("returned", "bool", nullable=True),
]

def validate(df, contract) -> None:
    """Fail the run at the boundary, with a message an on-call engineer can act on."""
    problems = []
    for spec in contract:
        if spec.name not in df.columns:
            problems.append(f"missing column: {spec.name}")
            continue
        actual = str(df[spec.name].dtype)
        if not actual.startswith(spec.dtype[:3]):
            problems.append(f"{spec.name}: expected {spec.dtype}, got {actual}")
        if not spec.nullable and df[spec.name].isna().any():
            n = int(df[spec.name].isna().sum())
            problems.append(f"{spec.name}: {n} nulls in a non-nullable column")

    extra = set(df.columns) - {s.name for s in contract}
    if extra:
        print(f"note: new upstream columns ignored: {sorted(extra)}")   # additive is fine

    if problems:
        raise ValueError("Contract violation in orders:\n  " + "\n  ".join(problems))

Two things to notice. Additive change is logged, not fatal — new columns should not wake anyone up. And the error message names the column and the observed value, because the person reading it at 3am is not the person who wrote it.

Data contracts, without the buzzword#

A data contract is the same idea made social: the producing team agrees, in writing and in code, to a schema and a set of guarantees, and agrees to version it rather than change it in place.

A useful contract states:

• Schema — columns, types, nullability.
• Semantics — what each field means, in a sentence a new hire could act on.
• Guarantees — freshness ("delivered within 30 minutes of the hour"), completeness ("no gaps > 5 minutes"), uniqueness ("order_id is unique").
• Change policy — additive changes any time; breaking changes require a new version and a deprecation window.
• An owner — a named team, not a person who has since left.

The contract does its real work socially. It converts "you broke my pipeline" into "the v2 contract was published in March with a June cutover", which is a conversation two teams can actually have.

Monitor distributions, not just schemas#

Structure passing tells you nothing about whether the data is right. Track, per load, per column:

• Row count, against the same weekday last week. A Tuesday that delivers 40% of a normal Tuesday is an incident even though every column is present.
• Null rate per column. A jump from 0.3% to 22% is a source change, every time.
• Distinct count on keys and categories. A new enum value appearing is worth a notification.
• Numeric distribution — min, max, median. A revenue column whose median jumped 100× is a currency-unit change, and you would like to know that before the finance team does.

Load dirty-customers into the Data Explorer to see what these checks are up against: five date formats, mixed currency units, and null represented five different ways in one column. Every one of those would pass a naive "column exists and is a string" check.