Analysed an A/B test on 290,585 users (145,274 control / 145,311 treatment) to determine whether a new landing page drives a statistically significant change in conversion rate — using two-proportion z-test, chi-square test, confidence intervals, and power analysis. Result: fail to reject H₀ (p = 0.1897 > 0.05); the new page shows no significant improvement.
Built as part of a Data Science portfolio targeting DS internships at product & tech companies.
Conversion rate optimisation is one of the highest-leverage activities in e-commerce. A 1% absolute lift in conversion rate on a platform with 290K users translates directly to revenue. Before shipping a new landing page to all users, companies run controlled A/B experiments to determine whether any observed difference is real or just noise.
This project replicates a real product team's decision workflow: formulate a hypothesis, run statistical tests, compute confidence intervals, assess power, and make a data-driven ship/no-ship recommendation.
E-Commerce A/B Testing 2022 — Download from Kaggle
| Column | Description |
|---|---|
| user_id | Unique user identifier |
| timestamp | Time of visit |
| group | control or treatment |
| landing_page | old_page or new_page |
| converted | 1 = converted, 0 = did not convert |
After cleaning:
- Raw rows: 294,480
- Dropped 3,893 mismatched rows (wrong page for group)
- Dropped 2 duplicate user_id rows
- Clean dataset: 290,585 rows
- Overall conversion rate: 11.9597%
Raw CSV is not committed to this repo. Download from the Kaggle link above and place as
data/ab_data.csv.
ABTestAnalysis/
├── data/ ← place ab_data.csv here (not committed)
├── notebooks/
│ └── ab_test_analysis.ipynb ← main notebook — run top to bottom
├── src/
│ ├── data_loader.py ← load & validate dataset
│ ├── statistical_tests.py ← all statistical test functions
│ └── visualisations.py ← all Plotly chart functions
├── plots/ ← all Plotly outputs (.html + .png)
├── .gitignore
├── pyproject.toml
└── README.md
Two-tailed test at α = 0.05:
- H₀: p_treatment = p_control — no difference in conversion rates
- H₁: p_treatment ≠ p_control — there is a difference
Two independent tests applied for robustness:
- Two-proportion z-test — directly tests equality of two proportions
- Chi-square test of independence — tests whether group membership and conversion outcome are independent
95% CI computed via normal approximation for the difference in proportions (treatment − control).
Observed statistical power calculated using Cohen's h effect size and statsmodels. Required sample size computed for 80% power at the observed effect size.
| Group | Users | Conversions | Conversion Rate |
|---|---|---|---|
| Control | 145,274 | 17,489 | 12.04% |
| Treatment | 145,311 | 17,264 | 11.88% |
- Absolute lift: -0.1579 pp (treatment is slightly lower)
- Relative lift: -1.31%
| Test | Statistic | p-value | Significant? |
|---|---|---|---|
| Two-proportion z-test | z = -1.3116 | 0.1897 | ✗ No |
| Chi-square test | χ² = 1.7054 | 0.1916 | ✗ No |
Both tests agree: fail to reject H₀.
95% CI for difference (Treatment − Control):
Lower bound : -0.3939 pp
Upper bound : +0.0781 pp
Margin of error : ±0.2360 pp
The CI contains zero — we cannot confidently say the treatment is better or worse than control.
| Metric | Value |
|---|---|
| Effect size (Cohen's h) | 0.0049 |
| Observed power | 25.89% |
| Required N per group (80% power) | 662,885 |
The observed power of 25.89% is well below the 80% threshold — the test lacked statistical sensitivity to reliably detect an effect this small.
Do not ship the new landing page.
Both the z-test (p = 0.1897) and chi-square test (p = 0.1916) fail to reach significance at α = 0.05, and the 95% CI [-0.39pp, +0.08pp] crosses zero — meaning we cannot rule out that the observed -0.16pp difference is due to chance. The observed power of only 25.89% indicates the test was severely underpowered: to reliably detect an effect this small (Cohen's h = 0.0049) at 80% power, we would need 662,885 users per group — more than 4× the current sample size.
Recommended next steps:
- Run the test for a longer duration to accumulate more power
- Test a more impactful design change rather than a subtle variant
- Perform segment-level analysis (by device, geography, or user tenure) to check for heterogeneous treatment effects that the aggregate result may be masking
This project uses uv for dependency management.
# 1. Clone the repo
git clone https://github.com/meet7364/ABTestAnalysis.git
cd ABTestAnalysis
# 2. Install dependencies
uv sync
# 3. Place the dataset
# Download ab_data.csv from Kaggle and place it in data/
# 4. Open the notebook
uv run jupyter notebook notebooks/ab_test_analysis.ipynbDependencies: pandas, numpy, scipy, statsmodels, plotly, kaleido
Meet Modi — B.Tech CSE (Data Science), VIT Chennai
LinkedIn · GitHub