When not to build a model

Model projects are expensive: data assembly, training, validation, serving, monitoring, retraining, and someone owning it for years. A model that saves less than it costs to keep alive is a liability with a dashboard.

Here is how to tell before you start.

Do not build a model when…#

Nobody will act on the prediction. The most common failure. If there is no intervention attached — no retention offer, no dispatch, no price change — a churn score is a number people look at and feel informed by. Find the action first, or do not start.

A rule performs almost as well. Try the rule. Genuinely try it, and measure it on the same split.

# The "model" a stakeholder can explain in a sentence.
high_risk = (df["days_since_login"] > 21) & (df["tenure_days"] > 60)

If that captures 70% of what a gradient-boosted model captures, ship the rule. It requires no serving infrastructure, no monitoring, no retraining, and no explanation. You can change it in an afternoon when the business changes, which a model cannot do.

The base rate is tiny and the cost of a false positive is high. At a 0.1% base rate, even a very good model produces mostly false positives at any usable recall. Do the arithmetic with a confusion matrix before promising anything.

The data does not exist yet. If the signal you need is not being logged, the project is an instrumentation project. Say that. It is a real and valuable finding, and it is much cheaper to hear in week one than in month four.

The relationship is unstable. If the pattern changes faster than you can retrain — fashion, virality, an adversarial setting — offline validation will look fine and production will not.

The question is causal. "What will happen if we raise prices?" is not a prediction problem. A model trained on historical prices learns who bought at which price, not what happens when you change it. See module seven.

The alternatives, roughly in order of preference#

A well-chosen threshold on one variable. Astonishingly often enough. Use the Distribution Lab or a simple percentile to pick it.

A scorecard. Three to six weighted rules, hand-tuned with domain experts. Explainable, adjustable, auditable, and it survives a data outage. Widely used in credit and fraud for exactly these reasons.

A lookup table. Segment by two or three dimensions and compute the historical rate per cell. This is a model — a fully interpretable one — and it beats a complicated model surprisingly often when data per cell is adequate.

Better instrumentation. Sometimes the answer is that you cannot predict the thing, and the correct output of the project is "here is what we would need to log".

A forecast baseline. Seasonal-naive, or a simple exponential smoothing, before anything with a training loop.

The cost side of the ledger#

Before proposing a model, price it honestly:

• Data assembly and feature pipelines: usually the majority of the work.
• Training and validation: the visible part, and the smallest.
• Serving: an API, or a batch job writing scores to a table.
• Monitoring: input drift, output drift, and the actual outcome once it arrives.
• Retraining: a schedule, a trigger, and a rollback path.
• Ownership: a named person, indefinitely.

Against that: the value of the decisions it improves. Estimate it before starting, even roughly. (cases per year) × (improvement in decision quality) × (value per case). If that number is smaller than a quarter of an engineer's time, you have your answer.

When you should build one#

To be clear — models earn their keep when:

• The decision is repeated at volume. Thousands of times a day, not four times a quarter.
• The pattern is genuinely complex — many interacting features, non-linear.
• The action is clear and its value is measurable.
• The feedback loop closes: you find out whether the prediction was right, so you can retrain and measure.
• Marginal accuracy has real value — a percentage point is worth something.

Fraud detection, demand forecasting, recommendations, ad ranking, credit scoring. High volume, clear actions, closed loops.

The version that usually wins#

When a model is justified, the sequence that works:

1. Rule baseline. Measure it.
2. Logistic or linear regression with well-engineered features. Often within a few points of anything fancier.
3. Gradient boosting with defaults. Usually the practical ceiling for tabular data.
4. Stop.

Step four is the hard one. The gap between step 3 and an extensively tuned ensemble is typically one to three points of AUC, and it costs weeks and a permanent maintenance burden. Spend those weeks on features, on the threshold, or on the intervention the prediction triggers — all three usually pay better.