The churn model that broke its own evaluation

We shipped a churn model. It scored well offline. Retention offers went to the top decile.

Three months later, measured performance had collapsed. Precision in the top decile was far below validation.

The model was not broken. It was working. Accounts flagged as high risk got a retention offer, the offer worked often enough, and they did not churn. So the label we were scoring against — "did they churn" — had been changed by the prediction itself.

This is a feedback loop, it is completely predictable in hindsight, and it invalidates naive performance measurement entirely. Any model whose output triggers an intervention that affects the outcome has this. Churn, fraud, credit, demand forecasting that drives inventory that constrains sales.

The only real fix is a permanent holdout: a random slice of high-risk accounts that never receives the intervention, so you retain a clean measurement of what would have happened. That costs money — literally, in churned accounts you could have saved — and it is the price of knowing whether the system works.

We argued about the holdout size for a week. We settled on 5%, which is enough to measure and small enough to justify. The argument itself was useful: it forced the retention team to say out loud what the offer was worth, which nobody had quantified.

The other thing we added: monitoring the decision distribution, not only the score distribution. What fraction of accounts crossed the threshold this week? That number has an operational consequence — it is the retention team's workload — and it moves for reasons the score distribution does not show.