The offline–online gap

Every model performs worse in production than in validation. A small gap means your validation was honest. A large one means something specific is wrong, and there are only about seven candidates.

Here they are, ordered by how often I have found each to be the culprit.

1. Leakage (most common by a distance)#

A feature that was not actually available at prediction time. The model learned from the future.

The specific forms:

• Target leakage. A field populated as a consequence of the outcome. cancellation_reason in a churn model. days_to_resolution when predicting resolution time.
• Backfilled fields. A column that is null when the row is created and filled in later. In your training snapshot it is populated; at scoring time it is null.
• Aggregate leakage. A feature computed over the whole dataset rather than as-of the prediction date.

The diagnostic: for every feature, write down the timestamp at which its value becomes known. If any is after your prediction time, that is your gap. Do this as a table in the model's documentation — it takes twenty minutes and it is the highest-value twenty minutes in the project.

The tell: an offline metric that is suspiciously good. An AUC of 0.97 on a churn problem is not a triumph, it is a bug report.

2. Validation that did not match deployment#

You random-split a time series. Or the same user appeared in train and test. Or you tuned hyperparameters on the same folds you reported from.

The rule: the split must simulate the situation the model will be in. Time-ordered problems get forward chaining with a gap matching your real prediction lag. Grouped data gets grouped splits. Anything tuned needs either nested CV or a test set touched exactly once.

3. Training–serving skew#

The features are computed by different code in training and serving, and the two disagree.

This is depressingly common: a pandas transformation in the training notebook and a hand-written SQL or Java equivalent in the serving path. One rounds, the other truncates. One treats null as zero, the other drops the row. The model sees different inputs than it was trained on and nobody notices because both pipelines "work".

The fix: one implementation, used by both paths. A feature store does this; so does a shared library; so does generating both from the same SQL. What does not work is discipline alone.

The diagnostic: log the feature vector at serving time for a sample of requests, recompute the same rows through the training pipeline, and diff them. Do this before launch, not after the incident.

4. Distribution shift#

The world changed. New market, new season, new marketing campaign that changed who shows up, a competitor's launch.

Monitor input distributions from day one — PSI or a simple quantile comparison per feature. This is the only cause on this list that gets worse over time rather than being present at launch, so it is the one that explains a model that started fine and decayed.

5. The feedback loop#

The model changes the thing it predicts. A churn model triggers retention offers, which prevent churn, which makes the model look wrong.

This is not a failure — it is the model working. But it means naive performance measurement is now invalid, because the labels you are scoring against were affected by the predictions. You need a holdout group that never receives the intervention, permanently. That costs real money and it is the only way to keep measuring.

6. The threshold moved, or was never set properly#

The model is fine and the decision rule is wrong. A threshold tuned on a validation set with a different class balance will produce a different volume of positives in production, and if a human team is acting on those positives, their capacity is now the binding constraint.

Check the decision distribution, not just the score distribution: what fraction of cases cross the threshold today versus in validation?

7. Small-sample optimism#

Your validation set was 400 rows and you picked the best of twelve model variants. The winner's score includes the luck that made it the winner.

The fix is fold variance: report 0.741 ± 0.028 rather than 0.741. If two models are within a fold standard deviation of each other, they are indistinguishable, and you should ship the simpler one.

The pre-launch checklist#

Before any model goes live:

• Every feature has a documented as-of timestamp, and none is after prediction time.
• The validation split matches deployment: time-ordered, grouped, gap included.
• Serving features have been diffed against training features on a real sample.
• A baseline — a rule, or last-value — has been measured on the same split.
• Input and prediction distributions are logged from day one.
• The threshold has been chosen from the cost asymmetry, and the resulting decision volume is something the downstream team can absorb.
• A holdout exists if the model's action affects the outcome.

And after launch, measure the gap deliberately. It is a property of your process, not a surprise. A team that knows its typical offline–online gap is 3 points can plan around it. A team that is surprised every time has not yet learned what its validation is worth.