How to use the cohen's kappa calculator
- Enter the counts as a square table, one row per line. Rows are rater A's categories and columns are rater B's categories, so the number in row 2, column 3 counts the items A put in category 2 and B in category 3.
- Choose Weighting. Leave it as none for unordered categories. Choose linear or quadratic only when the categories are in order and a near miss deserves partial credit.
- Read kappa, the observed agreement and the agreement expected by chance. For unweighted kappa, the standard error and 95% interval are also shown.
- Check the category table to see where the raters agree most.
Formula
observed agreement p_o = (items on the diagonal) / N
chance agreement p_e = sum over categories of (row total x column total) / N^2
kappa = (p_o - p_e) / (1 - p_e)
weighted kappa = 1 - (weighted observed disagreement) / (weighted chance disagreement)
For unweighted kappa the standard error is the square root of p_o (1 - p_o) / (N (1 - p_e)^2). Linear weights use |i - j| / (k - 1) for the disagreement between categories i and j, and quadratic weights use the square of that.
Two raters sorting 80 items into three categories
The table has rows 22, 2, 2; 5, 18, 3; and 1, 4, 23. Agreement on the diagonal is 63 of 80, so the observed agreement is 0.7875. The agreement expected by chance from the row and column totals is 0.33375, so kappa = (0.7875 - 0.33375) / (1 - 0.33375) = 0.6811. The standard error is 0.0686 and the 95% interval runs from 0.547 to 0.816. Linear weights give 0.7230 and quadratic weights give 0.7636.
For the classic two by two table 20, 5; 10, 15, the observed agreement is 0.70, chance agreement is 0.50 and kappa is 0.4. Perfect agreement gives 1, and tables where the two raters' choices are unrelated give 0.
Why kappa instead of percent agreement
Percent agreement counts matches but ignores that some matches happen by luck. If both raters call 90% of items "yes", they agree on at least 81% of items even if they pick at random. Kappa subtracts the agreement expected by chance from both the numerator and the maximum, so 0 means agreement no better than chance and 1 means perfect agreement.
Kappa also depends on how common each category is. The same observed agreement can give a low kappa when one category dominates. For that reason it is wise to report the table and the observed agreement along with kappa.
Reading the number
You will find published guidelines that label kappa ranges as slight, moderate or good. They are conventions rather than laws and different fields draw the lines in different places, so this page shows no labels. The highest kappa the row and column totals allow is also given, which tells you how much of any shortfall comes from the raters using the categories at different rates.
For two by two decision tables with accuracy, precision and recall, use the Confusion Matrix Calculator, and for the reliability of a multi-item scale the Cronbach's Alpha Calculator.
More statistics tools
For score scales, sampling, runs and robust averages, try the Robust Center Calculator, Standard Score Converter and Stratified Sample Allocation Calculator.
Assumptions and limits
- Two raters each rated every item once, using the same set of categories.
- Items are independent of each other.
- Counts are whole numbers and the table is square.
- The standard error and interval use the simple large-sample formula and are shown only for unweighted kappa.
- The example table is an illustration. Replace it with your own counts.
Frequently asked questions
Can kappa be negative?
Yes. A negative value means the raters agree less often than chance would produce, which usually points to a mix-up in the categories or a problem with the rating process.
Why is kappa low when the raters agree 90% of the time?
If almost every item falls in one category, chance agreement is also high, so there is little room above it. Kappa then drops even though the raw agreement looks excellent.
When should I use weighted kappa?
Use it when categories are ordered, like mild, moderate and severe, so that disagreeing by one step is less serious than by two. Quadratic weights punish big gaps most and give values close to a correlation.
How many items do I need?
The interval shown narrows as the number of items grows. Small samples give wide intervals, so with a few dozen items treat the value as a rough guide only.