6,000 rows · 8 columns

Checkout A/B test

Six thousand users randomised across a control and two variants, with conversion, revenue, device, and a novelty effect that fades over the first week.

Download CSV 303 KB Open in Explorer Query it in SQL

What is in it, and what is wrong with it

Variant B genuinely converts better. Variant A looks better in week one and then regresses — a novelty effect, and the single most common way a shipped experiment fails to replicate. Revenue per user is zero-inflated and heavy-tailed, so a t-test on revenue is not the right tool even though it will happily give you a p-value.

Schema

Column	Type	Description
user_id	string	User identifier.
variant	category	control, variant_a, or variant_b.
exposed_on	date	First exposure date.
device	category	desktop, mobile, or tablet.
sessions	integer	Sessions during the experiment window.
converted	boolean	Whether the user completed checkout.
revenue_usd	float	Revenue attributed to the user. Zero if not converted.
is_returning	boolean	Whether the user had visited before exposure.

First twelve rows

user_id	variant	exposed_on	device	sessions	converted	is_returning
U300000	variant_a	2024-03-12	desktop	2	false	false
U300001	variant_b	2024-03-28	mobile	3	false	false
U300002	variant_b	2024-03-26	mobile	3	false	false
U300003	control	2024-03-15	mobile	5	false	true
U300004	control	2024-03-16	mobile	4	false	true
U300005	variant_a	2024-03-25	desktop	2	false	false
U300006	control	2024-03-10	mobile	2	false	false
U300007	variant_a	2024-03-06	mobile	3	false	false
U300008	control	2024-03-19	desktop	4	false	false
U300009	variant_a	2024-03-21	tablet	2	false	false
U300010	variant_a	2024-03-04	desktop	3	false	false
U300011	variant_b	2024-03-13	tablet	3	false	false

Grain

One row per exposed user.

Things to try

Compute the conversion lift with a proper confidence interval.
Split by exposure week and watch variant A decay.
Try a t-test on revenue, then a bootstrap, and compare the intervals.
Check for sample ratio mismatch before you trust any of it.

Load it

python

import pandas as pd

df = pd.read_csv(
  "https://datanemics.com/datasets/data/ab-test-checkout.csv"
)

Licence

Public domain, CC0 1.0. Synthetic data — no attribution required.