No runbook, no page

New rule, adopted after a bad quarter: if it can page someone, it has a runbook. If it has no runbook, it is downgraded to a ticket.

No exceptions, and the enforcement is automated — the alert config validates against a runbook path that must exist.

The immediate effect was that about a third of our paging alerts got downgraded, because nobody wanted to write the runbook. That is exactly the right outcome. An alert nobody will document is an alert nobody has thought about, and it will be dismissed at 3am anyway.

The runbooks themselves are short and follow one shape:

# Runbook: orders ingestion

**What it does.** Hourly pull into `raw.orders`, then dbt builds
`fct_order_lines`. SLA: complete for the previous day by 08:00.

**Common failures**
1. Vendor 503 — transient, retries handle it. Safe to leave until
   morning; the lookback window picks up missed rows.
2. Schema validation failure — do NOT bypass. Read the diff in the
   failure message, patch staging, data backfills.
3. Duplicate key — usually a double delivery. Safe to rerun the day.

**Rerun one day:** dbt run --select fct_order_lines --vars '{"run_date": "..."}'

**What NOT to do.** Do not truncate `raw.orders`. It is the only copy;
the vendor's API only serves 90 days.

That last section is the one people forget and the one that prevents disasters. Two of our runbooks exist almost entirely for their "what NOT to do" line.

Overnight escalations roughly halved. Some of that is fewer alerts. Most of it is that the person on call can now resolve things themselves instead of waking whoever built it.