Data contracts without the ceremony

"Data contracts" arrived as a genuinely good idea and immediately acquired a layer of vendor ceremony: registries, schema servers, governance boards, and a lot of diagrams with arrows.

The underlying idea needs none of that. Here is the version I actually use.

The problem it solves#

Your pipeline breaks because a producing team changed something and did not know you existed. That is the whole problem. It is a communication failure with a technical symptom, which is why purely technical solutions keep half-working.

A contract does two things: it writes down what the consumer depends on, and it makes breaking that visible to the producer before it reaches production.

What goes in one#

Five things. It fits on a page.

Schema. Columns, types, nullability. The mechanical part.

Semantics. What each field means, in a sentence a new hire could act on. This is the part people skip and the part that prevents the expensive failures — the ones where the schema is fine and the meaning changed.

Guarantees. Freshness ("delivered within 30 minutes of the hour"), completeness ("no gaps longer than 5 minutes"), uniqueness ("order_id is unique per row"), volume ("between 40k and 90k rows on a weekday").

Change policy. Additive changes any time. Breaking changes get a new version and a deprecation window. State the window in weeks.

Owner. A team, with an escalation path. Not a person, because people leave.

dataset: orders_events_v2
owner: payments-platform
consumers: [analytics-eng, fraud-ml]
grain: One row per order state transition.
freshness: 30 minutes
volume: {weekday_min: 40000, weekday_max: 90000}
schema:
  - {name: order_id,     type: string,    nullable: false, unique_per: [order_id, state, occurred_at]}
  - {name: state,        type: string,    nullable: false, allowed: [created, paid, shipped, cancelled, refunded]}
  - {name: occurred_at,  type: timestamp, nullable: false, tz: UTC}
  - {name: amount_minor, type: integer,   nullable: false, note: "Minor units. Cents for USD. NOT dollars."}
  - {name: currency,     type: string,    nullable: false, allowed_pattern: "^[A-Z]{3}$"}
change_policy:
  additive: anytime
  breaking: new version, 8 week deprecation window

Note amount_minor and its note. That one line prevents the single most expensive class of bug in payments data, and no schema type system can express it.

Where it runs#

Two places, and both are needed.

In the producer's CI. A test that fails their build if the emitted data violates the contract. This is what makes it a contract rather than a document — the consequence lands on the person making the change, at the moment they make it.

At your ingestion boundary. A validation step that fails the load loudly if the contract is violated. This is your backstop for when the producer's CI is bypassed, which it will be.

If you only get one, take the second. It protects you. But the first is what actually changes behaviour, because it moves the cost of a breaking change to the person who is in a position to avoid it.

What it does not solve#

Semantic drift. status = 'active' used to include trials and now does not. Schema unchanged, types unchanged, contract satisfied, numbers wrong. The only defences are the semantics field above (so at least there is something to point at) and distribution monitoring (so you notice the proportion moved).

Data quality within the contract. A field can be present, correctly typed, non-null, and full of nonsense. Contracts check shape. Tests check content. You need both.

Organisational will. If the producing team has no incentive to care, a YAML file will not create one. The contract is a tool for teams that want to cooperate and lack a mechanism. It is not a substitute for the wanting.

The unglamorous version that works#

I have seen elaborate contract platforms fail and this succeed:

1. A YAML file in the consumer's repository, one per critical upstream dataset. Start with three; you do not have twenty critical upstreams.
2. A validation step at ingestion that fails loudly and names the violated field.
3. A pull request opened against the producer's repository adding a test that reads your YAML. Do the work of writing it yourself — that is the price of admission.
4. A Slack channel with both teams in it, where breakages get discussed rather than escalated.

Step four does more than steps one to three combined. Every serious data quality improvement I have been part of was preceded by two teams starting to talk to each other regularly, and the artefacts were how they had the conversation.

The starting point#

Take your three most load-bearing upstream datasets. Write the five fields for each. Put the validation at ingestion. Send the producers a link and ask if it looks right.

That is a morning's work and it will catch things this quarter. Everything above it in sophistication is optional and most of it stays optional for longer than the vendors suggest.