When should I not build a model?

A question that came in more than once, so the answer goes here.

When should I not build a model?

When a rule gets you most of the way, when you cannot get labels, when nobody will act on the prediction, or when being wrong is expensive and unexplainable.

The most common one is the first. A model that beats a sensible baseline by two points, and costs a deployment pipeline, a monitoring story and someone's ongoing attention, has not obviously won.

Compute the baseline first. Always. Predict the mean, predict last week, predict the majority class. If your model does not comfortably beat that, the finding is that the problem is hard or the features are wrong — and either is more useful than a mediocre model in production.

The fifth case, which is the most common and the least discussed: the decision the prediction feeds into is already made by a rule, and nobody is willing to change it. A model that is better than the rule but does not replace it has cost you a quarter and changed nothing. Establish what happens with the output before building the thing that produces it.

When not to build a model is the long version.