What is deliberately wrong with ab-test-checkout

What is deliberately wrong with ab-test-checkout.

6,000 rows, 8 columns, one row per exposed user. Six thousand users randomised across a control and two variants, with conversion, revenue, device, and a novelty effect that fades over the first week.

Variant B genuinely converts better. Variant A looks better in week one and then regresses — a novelty effect, and the single most common way a shipped experiment fails to replicate. Revenue per user is zero-inflated and heavy-tailed, so a t-test on revenue is not the right tool even though it will happily give you a p-value.

Something to try: Split by exposure week and watch variant A decay. The SQL playground has it loaded already, and the data explorer will profile every column in one pass if you would rather look before you query.

select *
from ab_test_checkout
limit 20;

The file is generated from a fixed seed, so the CSV you download today is byte-identical to the one from last year. That matters more than it sounds: it means a lesson can say "row 412 is the interesting one" and still be right in eighteen months.

Download it, break it, keep it. CC0 — no attribution required, no account, no email.