CUPED gave us back 40% of our sample size for ten lines of code

If you have each user's metric from before the experiment, you can use it to remove predictable variance from the metric during the experiment. Less variance means more power at the same sample size.

That is the whole idea. The implementation:

import numpy as np

def cuped_adjust(y, y_pre):
    """Variance-reduced version of y, using the pre-period covariate y_pre."""
    theta = np.cov(y, y_pre)[0, 1] / np.var(y_pre)
    return y - theta * (y_pre - np.mean(y_pre))

Run your usual analysis on the adjusted values instead of the raw ones. The expected difference between arms is unchanged — this is not a thumb on the scale — but the variance is lower, so the interval is tighter.

On a metric with strong pre-period correlation, we saw variance drop about 40%, which is roughly equivalent to a 40% larger sample. For a team that cannot get more traffic, that is enormous.

Three things to know before using it:

It needs pre-period data per user. If your metric is only defined post-exposure, or your users are new, it does nothing. Sessions-per-week works well; first-purchase conversion does not.

The correlation is what buys you the reduction. Weakly correlated covariate, negligible benefit. Check the correlation before assuming.

Decide to use it before you look at the results. Adjusted and unadjusted give slightly different answers, and choosing after the fact is choosing the one you like.

The related trick at assignment time is stratified randomisation on the same pre-period variable, which achieves something similar without post-hoc adjustment. Both are underused, and both are free power.