Checkout Redesign — A/B Experiment Analysis

An end-to-end experimentation project: from pre-registered power analysis through hypothesis testing, confidence intervals, guardrails, and segment analysis to a ship/hold decision and an executive summary.

The dataset is a documented simulation (data/simulate.py) with a known ground-truth treatment effect, so the analysis can be validated against truth — the test suite asserts the pipeline recovers the injected effect within its confidence interval and reaches the correct decision. Point it at a real experiment log with the same schema to analyze production data.

Business question

An e-commerce team tests a streamlined one-click checkout (treatment) against the existing multi-step checkout (control), 50/50 randomized.

• Primary metric: purchase conversion rate.
• Guardrails: revenue per user (must not drop) and checkout-error rate (must not rise).
• Segments: new vs returning users × mobile vs desktop.

Headline result (seed 42, 40,000 users/arm)

Decision	SHIP
Conversion	11.45% → 12.62%
Absolute lift	+1.17pp (95% CI 0.71–1.62pp), p = 4.1e-07
Relative lift	+10.2% (bootstrap 95% CI 6.0–14.4%)
Achieved power	99.9% · sample-ratio check: PASS
Guardrails	revenue no drop, errors no rise
Heterogeneity	lift concentrates in new users (+1.4 to +1.8pp); returning/mobile not significant

Full write-up: reports/summary.md (generated).

Method

• Power / sample size (src/ab/power.py): required n per arm, minimum detectable effect, achieved power — set before looking at outcomes.
• Hypothesis tests (src/ab/stats.py): two-proportion z-test (pooled test statistic, unpooled Wald CI), Welch's t-test for the continuous guardrail, and a percentile bootstrap for the relative-lift CI. Validated against scipy.
• Sanity checks (src/ab/experiment.py): sample-ratio-mismatch (chi-square).
• Segment analysis: per-segment lifts with CIs, flagged as uncorrected for multiple comparisons.
• Decision rule: ship only if the primary lift is significant and positive and no guardrail is breached.

Run it

pip install -r requirements.txt
python data/simulate.py --n_per_arm 40000 --seed 42   # -> data/experiment.csv
python data/load_to_sqlite.py                          # -> data/experiment.db
python -m src.ab.report --db data/experiment.db        # -> reports/summary.md
pytest -q                                              # validate against ground truth
streamlit run app/dashboard.py                         # interactive view

Extraction is done in SQL (sql/extract.sql) against SQLite; the Python layer only does the statistics and reporting.

Using real data

Provide a CSV with columns user_id, arm, user_type, device, converted, revenue, checkout_error (arm ∈ {control, treatment}) and skip the simulate step — load_to_sqlite.py and the analysis are dataset-agnostic.