How to use the p value adjustment calculator
- Paste your p values, one per line. You can put a label before a comma, such as "Group 2 vs control, 0.012".
- Enter your significance level (alpha), the cutoff you chose before looking at the data. The 0.05 shown is an example.
- Read the table of adjusted p values for six methods, side by side with the raw p value.
- Count how many remain at or below alpha under each method, shown above the table.
Formula
Bonferroni: p x m, capped at 1
Sidak: 1 - (1 - p)^m
Holm: sort ascending, p(i) x (m - i + 1), then keep the running maximum
Hochberg: sort descending, p(i) x (m - i + 1), then keep the running minimum
Benjamini-Hochberg: p(i) x m / i, keep the running minimum from the largest; Yekutieli also multiplies by 1 + 1/2 + ... + 1/m
Here m is the number of tests and i is the rank of the p value from the smallest. Adjusted values are capped at 1.
Five comparisons against a control
The p values 0.001, 0.012, 0.027, 0.041 and 0.30 give four results below 0.05 before any adjustment. Bonferroni multiplies by 5, giving 0.005, 0.06, 0.135, 0.205 and 1, so one result survives. Holm gives 0.005, 0.048, 0.081, 0.082 and 0.30, so two survive.
Benjamini-Hochberg gives 0.005, 0.03, 0.045, 0.05125 and 0.30, so three results are at or below 0.05 when you control the false discovery rate rather than the chance of any false positive. Benjamini-Yekutieli, which allows any dependence between tests, gives 0.0114, 0.0685, 0.10275, 0.117 and 0.685, so one survives.
Why adjust at all
If you run many tests, some will come out significant by chance alone. With 20 independent tests at the 0.05 level, you expect about one false positive even if nothing is going on. Adjusting the p values keeps the overall error rate where you want it.
Bonferroni, Sidak, Holm and Hochberg control the chance of any false positive (the family-wise error rate). The Benjamini methods control the expected share of false positives among the results you call significant (the false discovery rate), which is a weaker guarantee that suits screening many candidates.
Choosing a method
Decide the method before you see the data. Holm is never less powerful than Bonferroni and has no extra assumptions. Sidak and Hochberg assume the tests are independent or positively related. For the p values themselves, see the P Value Calculator, and for comparing several group means in one test, the ANOVA Calculator.
More statistics tools
For related statistical tests and measures, try the Cohen's d Calculator, AUC ROC Calculator and Confusion Matrix Calculator.
Assumptions and limits
- The p values come from tests that were all planned before the data were seen.
- Sidak, Hochberg and Benjamini-Hochberg assume independent or positively dependent tests.
- Adjusted p values are capped at 1 and compared with alpha at or below.
- The number of tests is the number of lines you enter.
Frequently asked questions
What is the Bonferroni correction?
It multiplies each p value by the number of tests, or equivalently divides alpha by that number. It is simple and safe but conservative when there are many tests.
Is Holm better than Bonferroni?
Holm controls the same family-wise error rate and is at least as powerful, so it finds as many or more significant results. It needs no extra assumptions.
What is the difference between FWER and FDR?
Family-wise error rate is the chance of one or more false positives. False discovery rate is the expected share of false positives among the results you call significant, which is more lenient.
What does an adjusted p value mean?
It is the smallest alpha at which that result would still be called significant after correcting for the number of tests, under the chosen method.