About the A/B Test Significance Calculator
An A/B test shows two versions of a page to different visitors and compares how many convert. The catch is that random luck alone can make one version look better. This calculator takes the number of visitors and conversions for each version and runs a two-proportion z-test. It shows both conversion rates, the relative uplift, the interval within which the true difference probably lies, the p-value and a plain verdict at the confidence level you pick. You can also enter the smallest lift you care about to see how many visitors per group you need. Everything runs locally in your browser.
Updated October 2026
How to use the A/B Test Significance Calculator
- 1Enter the visitors who saw version A and how many of them converted.
- 2Enter the same two numbers for version B.
- 3Pick a confidence level such as 95 percent.
- 4Optionally type the smallest lift you care about to get a sample size.
- 5Read the verdict, the p-value and the interval for the difference.
How the test works
The calculator pools both groups to estimate what the conversion rate would be if there were no difference, then asks how many standard errors apart your two rates are. That number is the z score, and the p-value is the chance of a gap that large or larger in either direction. A z score beyond 1.96 gives p below 0.05.
The interval shown is built from the two separate rates. If it excludes zero, the test agrees with the p-value. If it is wide, you need more data before making a decision.
Good habits for testing
- Pick one main metric before you start.
- Run the test for whole weeks so day-of-week effects balance out.
- Do not peek and stop early.
- Check that the traffic split is close to 50/50.
- Treat a tiny uplift with a huge sample as real but maybe not worth the change.
Related tools
For the size of the group you need, the Sample Size Calculator is a good companion, and the P-Value Calculator converts a test score to a probability. The Confidence Interval Calculator explains the interval idea on a single average, and the T-Test Calculator compares averages instead of conversion rates. Statistics support decisions but do not replace judgement.
Features
- Conversion rates and relative uplift
- Two-sided z-test p-value
- Confidence interval for the difference
- Choice of 90, 95 or 99 percent confidence
- Visitors needed per group for a chosen lift
- Clear checks for impossible numbers
- Free and private
Frequently asked questions
What does statistically significant mean?
It means a difference this large would be unlikely to appear by chance alone if the two versions really performed the same. At 95 percent confidence, the p-value must be below 0.05.
What is a p-value?
It is the probability of seeing a difference at least this big if there were no real difference. A small p-value is evidence against the idea that the versions are equal, but it is not the chance that B is better.
How many visitors do I need?
It depends on your base rate and the lift you want to detect. Detecting a 10 percent relative lift on a 5 percent conversion rate needs roughly 31,000 visitors per version. Fill in the lift box for your own figure.
Can I stop the test when it turns significant?
Stopping the moment you see significance inflates false positives. Decide the sample size first, or use a method designed for sequential testing.
What if the result is not significant?
It means there is not enough evidence yet. The versions may be the same, or the test may be too small to see a modest difference.
Is this a one-sided or two-sided test?
Two-sided, which asks whether B differs from A in either direction. It is the safer choice unless you decided in advance that only an improvement matters.
Does it work for click-through rate?
Yes, for any yes-or-no outcome per visitor, such as clicks, sign-ups or purchases. It does not suit averages like revenue per visitor.