Crash course / Modelling, honestly
Frame the problem, then beat a stupid baseline
Most model projects fail at framing, not at modelling. The second most common failure is not knowing what "good" would have been without a model at all.
The request arrives as "can we predict churn". That is not yet a modelling problem. Turning it into one is most of the work, and skipping it is why so many models get built and never used.
Framing questions, in order#
What decision does the prediction feed? If nobody will act differently, do not build it. A churn model with no retention offer attached is an expensive way to feel informed.
What is the unit and the horizon? "Will this account churn" is undefined. "Will this account cancel within 60 days of a given scoring date" is a problem you can build.
What is available at prediction time? The single most important question in the whole lesson, and the source of nearly every leakage bug. If your model scores accounts on the first of the month, it cannot use anything recorded after that date — including fields that get backfilled, like a cancellation_reason populated at cancellation.
What is the base rate? If 3% churn, a model predicting "nobody churns" is 97% accurate. Know this before anyone celebrates an accuracy number.
What are the costs of each error? A false positive sends an unnecessary discount to a happy customer: cheap. A false negative loses an account: expensive. That asymmetry sets your threshold, and the threshold matters far more than the model.
Classification, regression, or ranking?#
A surprising number of problems framed as classification are really ranking problems. "Which accounts should the retention team call this week?" does not need a calibrated probability of churn — it needs the top 200 accounts in order. That changes your metric from AUC or log-loss to precision@k, and it changes what "better" means.
Similarly, many regression problems are really threshold problems. If the action only depends on whether demand exceeds capacity, predict that, not the exact number.
The baseline is not optional#
Before any model, build the dumbest thing that could work and measure it properly. Your model must beat this, on the same split, on the same metric, or it has done nothing.
| Problem | Baseline |
|---|---|
| Classification | Predict the majority class. Then predict the base rate. |
| Regression | Predict the mean. Then the median. |
| Time series | Predict the last value. Then the value from the same period last week/year. |
| Ranking | Rank by popularity, or by the single most obvious feature. |
| Recommendations | Recommend the most popular items to everyone. |
The baseline is also your communication tool. "The model reduces error by 34% versus predicting last week's value" is meaningful to a stakeholder. "MAE is 412" is not.
Choosing a metric that means something#
Accuracy is almost always wrong for imbalanced problems. At a 3% base rate it rewards a model that never predicts the positive class.
Precision and recall trade off against each other, and you pick the trade-off from the cost asymmetry, not from a default.
F1 collapses that trade-off into one number by assuming precision and recall matter equally — which they almost never do. Use it for leaderboards, not for decisions.
ROC AUC is threshold-independent and interpretable as "probability a random positive scores above a random negative". Optimistic on heavily imbalanced data, because a large true-negative population flatters the false positive rate.
PR AUC is the better summary when positives are rare.
Log loss / Brier score for when you need the probability itself to be right — pricing, expected value calculations, anything where you multiply the probability by something.
MAE versus RMSE: RMSE punishes large errors more. Choose based on whether one 100-unit error is worse than ten 10-unit errors in your context. For skewed targets, consider MAPE's problems (it explodes near zero and is asymmetric) before reaching for it.
Leakage: the reason your offline numbers lie#
Leakage is when information unavailable at prediction time gets into training. It produces spectacular offline results and a model that fails on day one in production.
Common sources, all of which I have personally shipped at least once:
- Target leakage. A feature that is a consequence of the outcome.
cancellation_reasonwhen predicting cancellation.days_since_last_logincomputed after the churn date. - Temporal leakage. Random train/test split on time series, so the model trains on the future to predict the past.
- Aggregate leakage. Scaling or imputing using statistics computed over the whole dataset, so test-set information leaks into training via the mean.
- Group leakage. The same user appearing in both train and test. The model memorises the user rather than learning the pattern.
- Duplicate leakage. Near-duplicate rows split across train and test.
The check that catches most of it: for every feature, state the timestamp at which its value becomes known. If that timestamp is after your prediction time, it is leakage. Write this into the feature documentation.
The order of work#
- Frame the decision, the unit, the horizon, and the costs.
- Assemble the dataset with a strict as-of-time discipline.
- Build the baseline. Measure it on the real split.
- Build the simplest model that could beat it — regularised linear or logistic regression, or a single gradient-boosted model with default parameters.
- Only then consider anything more complex.
Most projects should stop at step four. A well-framed logistic regression with good features and an honest threshold beats a badly framed deep model in every dimension that matters, including the one where someone has to maintain it.