How to use the permutation test calculator
- Paste the values of Group A and Group B, each separated by commas, spaces or new lines.
- Choose whether to compare means or medians, and the alternative: a difference in either direction, A larger, or A smaller.
- Read the p value. It is the share of relabelings of the pooled values that give a difference at least as extreme as the one you observed.
- Open the advanced options to change the number of random relabelings or the seed. They matter only when the groups are too large to list every split.
Formula
observed difference d = statistic(A) - statistic(B)
Pool all n values and choose which k of them play the part of group A
p = (number of splits with a difference at least as extreme as d) / (number of splits)
Number of splits = n! / (k! (n - k)!)
Random version: p = (count + 1) / (draws + 1)
When there are 200,000 splits or fewer, every split is checked and the p value is exact. Otherwise the page draws random relabelings with a seeded generator, and the extra 1 counts the observed split so p is never 0.
Two groups of five
Group A is 12.1, 14.3, 13.8, 15.2, 16.0 with mean 14.28. Group B is 10.4, 11.9, 12.2, 13.1, 11.0 with mean 11.72. The observed difference is 2.56.
There are 10 choose 5 = 252 ways to split the ten values into two groups of five. Exactly 6 of them give a difference of 2.56 or more in size, so the two-sided p value is 6 / 252 = 0.0238. One-sided, with A larger, 3 splits qualify and p = 3 / 252 = 0.0119. Comparing medians, the difference is 2.4 and 12 splits are as extreme, so p = 0.0476.
The idea in plain words
If the group label made no difference, then swapping labels between values would change the observed gap only by chance. A permutation test does exactly that swapping, over and over, and counts how often chance alone produces a gap as large as yours. The answer is the p value. Nothing is assumed about bell curves, and it works with any statistic you can compute, here means and medians.
It sits next to the rank tests: Mann-Whitney and the Kruskal-Wallis test compare ranks, while this test compares the actual numbers. To describe how uncertain a single statistic is, the bootstrap interval resamples instead of relabeling.
Exact and random results
For two groups of 10 and 10 there are 184,756 splits, so the page checks them all. For 12 and 12 there are 2,704,156, so it draws random relabelings instead. The same seed always gives the same estimate, and the page shows the approximate size of the random error so you can judge whether to draw more.
Ties between values are handled by comparing differences with a tiny tolerance, so a split that ties the observed difference counts as at least as extreme.
Assumptions and limits
- The values could be swapped between the groups if the groups made no difference (exchangeability), which usually holds in a randomized comparison.
- The groups are independent of each other. Paired or repeated measurements need a paired method.
- Group sizes are small enough for the page to run: up to 400 values per group.
- The random estimate depends on the seed and the number of draws. The exact result does not.
Frequently asked questions
What is the difference between exact and random p values?
The exact value counts every possible split, so it never changes. The random value samples the splits, so it varies a little with the seed. Both estimate the same quantity, and the random one improves with more draws.
Why is the random p value never exactly 0?
The observed split is itself one possible relabeling. Adding 1 to the count and to the number of draws includes it, which keeps the estimate honest and avoids claiming that an event is impossible.
Can I compare medians?
Yes. Choose medians in the statistic list. The test then uses the difference in medians of the relabeled groups, which a rank test does not report directly.
Does a small p value prove the groups differ?
It says the observed gap would be unusual if labels were interchangeable. Study design, measurement quality and sample size still decide what that means. This page describes the numbers you entered.