How to use the a/b test significance calculator
- Enter the number of visitors and conversions for version A (the control) and version B (the variant).
- Pick a confidence level: 90%, 95% or 99%. The page tells you whether the difference clears that level.
- Open the advanced options to plan a test: set a baseline rate, the smallest lift you care about and the power.
- Read the p-value, the interval for the difference, the Fisher exact check and the visitors needed per version.
Formula
z = (rate B - rate A) / sqrt(p x (1 - p) x (1/nA + 1/nB)), with p the pooled rate
p-value (two-sided) = 2 x the upper tail of the normal curve beyond |z|
Interval = difference plus or minus z x sqrt(pA(1-pA)/nA + pB(1-pB)/nB)
Visitors per version = (z(alpha) x sqrt(2 x pbar x (1-pbar)) + z(power) x sqrt(p1(1-p1) + p2(1-p2)))^2 / (p2 - p1)^2
The Fisher exact test sums the probabilities of all tables at least as unlikely as yours with the same totals. It is a check that does not rely on the normal curve.
5,000 visitors per version
Version A converts 250 of 5,000 (5.00%) and version B converts 300 of 5,000 (6.00%). The difference is +1.00 points, a relative lift of +20%. The pooled rate is 5.5%, the z score is 2.1932, and the two-sided p-value is 0.0283 (one-sided 0.0141). At 95% confidence the difference is significant. The 95% interval for the difference is 0.107 to 1.893 points. The Fisher exact p-value is 0.0315, close to the normal result.
For planning, a baseline of 5% and a smallest lift of 10% (to 5.5%) at 95% confidence and 80% power needs 31,234 visitors per version. A smaller 5% lift needs 122,124 per version, a bigger 20% lift needs 8,158, and 90% power at the 10% lift needs 41,813.
What the p-value means
The p-value is the probability of seeing a gap at least this large between the two versions if there were no real difference. A small value is evidence that the gap is not just chance. It is not the probability that B is better, and it says nothing about how big or how useful the difference is. Always read the interval for the difference alongside it.
With 5,000 visitors per version, the interval above runs from 0.1 to 1.9 points. A lift near the low end might not be worth the cost of the change. For related tests, see the P-Value Calculator, the Chi-Square Calculator and the Confidence Interval Calculator.
Good habits for running a test
Decide the sample size before you start, using the planning section or the Sample Size Calculator, and avoid stopping early the moment the result looks significant. Looking at the results again and again and stopping at a lucky moment makes false positives much more common than the stated confidence level.
Split visitors at random, run both versions at the same time and measure the same thing in each. Several tests at once, or many variants, raise the chance that one looks significant by luck. This page is a statistical aid, and the decision to ship should also consider cost, risk and other evidence.
Assumptions and limits
- Each visitor is counted once and converts or does not. Visitors in each version are independent.
- The z test uses a normal approximation and works best when each version has at least 10 conversions and 10 non-conversions. The Fisher check helps with small counts.
- Planning uses a two-sided test with equal-sized versions and the baseline rate you enter.
- The page is informational. It does not decide whether a change is worth making.
Frequently asked questions
What does statistically significant mean?
It means the observed gap would be unlikely if the two versions had the same true conversion rate, at the confidence level you picked. It does not mean the effect is large or that B will always win.
Why does the page show a Fisher exact p-value too?
The z test uses a smooth curve that is only an approximation. The Fisher exact test counts all possible outcomes with the same totals, so it works for any counts and acts as a second opinion on the result.
Why is the sample size so large for small lifts?
The number of visitors needed grows with the square of how small the difference is. Cutting the lift in half needs about four times as many visitors per version.
What is power?
Power is the chance that the test detects a real difference of the size you set. At 80% power, a real lift of that size will be found as significant about four times in five.