How to use the intraclass correlation calculator
- Paste your ratings with one row per subject and one column per rater. Every row needs the same number of ratings.
- Pick the headline form you want to report and the confidence level.
- Read the headline value and its interval, then check the table for all six forms, their F ratios and p values.
- Choose the form by how your raters were selected and whether you need absolute agreement or only consistency.
Formula
MSR = subjects mean square, MSC = raters mean square, MSE = error mean square, MSW = within-subject mean square
ICC(1,1) = (MSR - MSW) / (MSR + (k - 1) MSW)
ICC(2,1) = (MSR - MSE) / (MSR + (k - 1) MSE + k (MSC - MSE) / n)
ICC(3,1) = (MSR - MSE) / (MSR + (k - 1) MSE)
ICC(1,k) = (MSR - MSW) / MSR; ICC(2,k) = (MSR - MSE) / (MSR + (MSC - MSE) / n); ICC(3,k) = (MSR - MSE) / MSR
n is the number of subjects and k the number of raters. Confidence limits follow McGraw and Wong (1996), using F distribution limits, and the ICC(2,1) limits use a Satterthwaite degrees of freedom adjustment.
Six subjects rated by four judges
This is the table from Shrout and Fleiss (1979): rows 9, 2, 5, 8; 6, 1, 3, 2; 8, 4, 6, 8; 7, 1, 2, 6; 10, 5, 6, 9; 6, 2, 4, 7. The mean squares are MSR = 11.2417, MSC = 32.4861, MSE = 1.0194 and MSW = 6.2639.
The single-rater values are ICC(1,1) = 0.1657, ICC(2,1) = 0.2898 with a 95% interval of 0.0188 to 0.7611, and ICC(3,1) = 0.7148. The average-of-four versions are 0.4428, 0.6201 and 0.9093. They match the published 0.17, 0.29, 0.71, 0.44, 0.62 and 0.91.
Choosing among the six forms
The first number in ICC(a,b) says how raters were chosen. Form 1 is for designs where each subject is rated by a different set of raters. Form 2 treats the raters as a random sample from a larger pool and measures absolute agreement. Form 3 treats these particular raters as the only ones of interest and measures consistency, ignoring a constant offset between raters.
The second number says whether you use one rater's score (1) or the average of all k raters (k). Averages are more reliable than single scores, so the k forms are always higher. The example shows how large the gap between forms can be: the same table gives 0.29 or 0.71 for a single rater depending on whether rater offsets count as disagreement.
Related agreement measures
For categories instead of scores, use Cohen's kappa. For comparing group means after splitting variance, see the ANOVA calculator, which uses the same mean squares. This page reports numbers and does not label them good or poor, because suitable cut points depend on the field.
Confidence intervals for small studies are wide, as the example shows. An interval from 0.02 to 0.76 says the data are compatible with very low and fairly high reliability.
Assumptions and limits
- Ratings are numeric, every rater rated every subject, and there are no missing cells.
- Subjects are a sample from a larger set, and the usual ANOVA assumptions of roughly bell-shaped errors with equal spread hold.
- Negative values can occur when disagreement within subjects exceeds the variation between subjects.
- The page needs at least 3 subjects and 2 raters, and refuses tables with zero error variance.
Frequently asked questions
Which form should I report?
Report the form that matches your design and say so with its full label, such as ICC(2,1) for absolute agreement among randomly chosen raters rated singly. Reporting only "ICC" leaves readers unable to tell which of the six values you mean.
Why can the ICC be negative?
The estimate subtracts a mean square of error from a mean square of subjects. If raters disagree about as much as subjects differ, the difference can fall below zero. It signals very poor reliability, not a calculation fault.
Why are the average-rater forms higher?
Averaging k ratings cancels part of each rater's random error, so the average is more reliable than any single rating. The Spearman-Brown relation connects the two: ICC(k) = k x ICC(1) / (1 + (k - 1) x ICC(1)).
Can I include missing ratings?
Not on this page. A subject with a missing cell has to be dropped or the table completed, because the mean squares assume a full subjects by raters grid.