How to use the beta-binomial posterior calculator
- Enter the successes and trials for option A and for option B, such as sign-ups and visitors.
- Set the prior. The values 1 and 1 are a flat starting belief. Larger numbers act like extra earlier data, and values must be at least 0.5.
- Choose the width of the credible interval.
- Read the chance that B beats A, the posterior mean and interval for each option, and the expected loss of picking either one.
Formula
posterior = Beta(prior successes + successes, prior failures + failures)
mean = alpha / (alpha + beta)
P(B > A) = integral of density_B(x) x cdf_A(x) over 0 to 1
expected loss of choosing B = E[max(pA - pB, 0)]
The integrals are computed numerically. Credible limits are found by inverting the regularized incomplete beta function.
Two options, 200 trials each
Option A has 45 successes in 200 trials and option B has 60 in 200, with a flat Beta(1, 1) prior. The posteriors are Beta(46, 156) and Beta(61, 141), with means of 22.77 percent and 30.20 percent. The 95 percent credible intervals are 17.27 to 28.79 percent and 24.08 to 36.69 percent.
The chance that B has the higher rate is 95.54 percent. If you pick B and A turns out better, you give up 0.0804 percentage points on average. If you pick A and B is better, you give up 7.506 percentage points. With 3 of 10 and 8 of 10 and a prior of 0.5 and 0.5, the chance that B is better is 98.89 percent.
What the numbers mean
A credible interval is a range that holds the true rate with the stated probability, given the prior and the data. The chance that B beats A is a direct statement about the two unknown rates, which is the number many people wish a significance test gave them. The expected loss turns the comparison into a decision: it is the average cost, in rate points, of choosing the option that turns out to be worse.
The result depends on the prior, so with little data the prior matters. A flat prior is the neutral choice. Tiny samples give wide intervals and a chance near 50 percent, as they should.
Related tools
For a classical test of the same two rates, use the A/B test significance calculator. For single probabilities and updating a belief with one piece of evidence, see the Bayes theorem calculator and the binomial probability calculator. To plan how much data you need, use the statistical power calculator.
Assumptions and limits
- Each trial is independent and has the same success rate within an option, and the two options are independent of each other.
- The prior is a Beta distribution with both numbers at least 0.5.
- Successes and trials are whole numbers, with successes not above trials.
- The tool does not decide when to stop collecting data. Stopping when the numbers look good can mislead.
Frequently asked questions
Is a 95 percent chance that B is better enough to act?
That depends on what a wrong choice costs. The expected loss helps: if it is tiny compared with what matters to you, acting is reasonable even when the chance is not extreme. A high chance with a large possible loss may still call for more data.
How is this different from a p value?
A p value is the chance of data this extreme if there were no difference. The chance reported here is the probability, given your prior and the data, that B has the higher rate. They answer different questions and need not agree.
How do I choose a prior?
Use 1 and 1 when you have no earlier knowledge. If you have earlier results, such as 20 successes in 100 trials, you can use 20 and 80 to carry them in, but then say so, since the result depends on that choice.
Why can the chance be exactly 50 percent?
With no data and identical priors, nothing separates the two options. The tool shows 50.00 percent in that case.