A/B Test Significance Calculator
Check whether a difference in conversion rates between two groups is statistically significant, using a two-proportion z-test.
Method & Limitations
This uses a two-proportion z-test with a pooled proportion — the standard method for comparing two conversion rates. The p-value is calculated from the standard normal distribution using a well-known numerical approximation (Abramowitz & Stegun 7.1.26), accurate to roughly 7 decimal places.
- The normal approximation is less reliable with very few conversions per group (a common rule of thumb is at least ~10 per group, ideally more). With small samples, gather more data or use an exact test instead.
- This checks ONE comparison at a single point in time. Repeatedly checking results as data arrives ("peeking") inflates the false-positive rate beyond the confidence level shown.
- Statistical significance is not the same as practical importance. A tiny, statistically significant lift may not be worth implementing; a larger, not-yet-significant lift may just need more data before you can rule out chance.
Worked Example
Group A: 1,000 visitors, 100 conversions (10%). Group B: 1,000 visitors, 125 conversions (12.5%). At a 95% confidence level, this particular example clears the bar for statistical significance — try it in the calculator above to see the exact z-score and p-value.