Monitoring data, not just jobs

Every orchestrator ships job monitoring: did the task succeed, how long did it take. That is necessary and it catches perhaps half of what actually goes wrong. The other half is jobs that succeed and produce wrong data — covered in the quality module as silent failures, and here is the operational side.

Four layers, and what each catches#

1. Job health. Did it run, succeed, and finish in a normal amount of time? Catches crashes and hangs. Every orchestrator gives you this free.

2. Freshness. Is the output recent enough? Catches the job that succeeded and loaded nothing, which job health cannot see.

3. Volume and distribution. Is the shape of today's data normal? Catches partial loads, upstream filters, and unit changes.

4. Semantic checks. Do the business relationships still hold? Catches the failures nothing else can — a revenue total that no longer reconciles, a funnel where step 3 exceeds step 2.

Most teams have layer one. Layer two is cheap and catches the most damaging failure mode. Layers three and four are where the mature teams differentiate.

Freshness, concretely#

create or replace view monitoring_freshness as
select 'fct_order_lines' as table_name,
       max(ordered_at)   as latest_event,
       max(_loaded_at)   as latest_load,
       datediff('minute', max(_loaded_at), current_timestamp) as minutes_stale,
       360                                                    as sla_minutes
from fct_order_lines
union all
select 'dim_customers', max(valid_from), max(_loaded_at),
       datediff('minute', max(_loaded_at), current_timestamp), 1440
from dim_customers;

Two distinct timestamps, deliberately. latest_event is the newest business event; latest_load is when your pipeline last wrote. A gap between them means the pipeline is healthy and the source has stopped — a different incident with a different owner, and one you want to identify before paging your own team.

Volume, compared against the right thing#

Compare like with like. Business data is weekly-seasonal; a Monday-versus-Sunday check will alert every week until someone silences it.

with daily as (
    select order_date, count(*) as rows_loaded
    from fct_order_lines
    where order_date >= current_date - 56
    group by 1
),
same_weekday as (
    select order_date, rows_loaded,
           avg(rows_loaded) over (
               partition by dayofweek(order_date)
               order by order_date rows between 8 preceding and 1 preceding
           ) as expected
    from daily
)
select *, rows_loaded / nullif(expected, 0) as ratio
from same_weekday
where order_date = current_date - 1
  and (rows_loaded < 0.6 * expected or rows_loaded > 1.6 * expected);

Start the thresholds wide. A monitor that fires constantly is worse than no monitor, because it trains people to dismiss the channel — and they will dismiss the real one along with the rest.

Distribution monitoring is what catches semantic drift#

Track, per load, per important column: null rate, distinct count, and for numerics the median and p99. Alert on the change, not the level.

This is the layer that would have caught the bnpl incident from the quality module. The share of rows falling into a case statement's else branch went from 0% to 4%, and no structural check can see that. A category-proportion monitor sees it on day one.

Monitoring models specifically#

Models degrade quietly, and there is a specific order to the signals:

Input drift arrives first. The distribution of features shifts. Detectable immediately, and usually the earliest warning you get.

Prediction drift next. The distribution of scores shifts. Also immediate, and a strong signal — if your average predicted churn probability jumped from 0.08 to 0.19 overnight, something upstream changed.

Performance decay last, and only once labels arrive — which may be 60 days later for a churn model. That delay is exactly why you monitor the first two.

def population_stability_index(expected, actual, bins=10):
    """PSI over 0.2 is the conventional line for 'investigate'."""
    import numpy as np
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

Also monitor the decision, not just the score: what fraction of accounts crossed your action threshold today? That is the number with an operational consequence, and it can move sharply even when the score distribution barely shifts.

Dashboards nobody looks at#

A monitoring dashboard is not monitoring. Nobody opens it on a good day, so it only gets read after someone already noticed the problem.

Push, do not pull. A daily digest in the channel where the team already lives — five lines, freshness, volume, test results, cost — gets read. A dashboard behind a login does not.

Incident hygiene#

When something breaks: communicate before you fix. A message saying "revenue dashboard is stale, we know, ETA 11:00" prevents six people investigating the same thing and prevents a decision being made on bad data in the meantime.

Then a short write-up. Not blame — a timeline, the root cause, and the control that would have caught it earlier. Half the time that control is a five-line test, and writing it is the actual deliverable of the incident.