Intraclass Correlation
Single-rater and average-rater intraclass correlation from an ANOVA mean-square table.
The reliability of one rater’s score — the figure to report when a single rater will be used in practice.
The reliability of the average across all 3 raters, which is always the higher of the two. Good.
These are the one-way random-effects forms, ICC(1,1) and ICC(1,k). Two-way models — used when the same raters assess every subject — need a different mean-square breakdown, so check which form your design calls for before reporting.
The ICC is the share of total variance that comes from real differences between subjects rather than from rater disagreement. With MSB = 12.4, MSW = 2.1 and 3 raters, the single-rater ICC is (12.4 − 2.1) ÷ (12.4 + 2 × 2.1) = .62, and the average of 3 raters gives .83.
Reliability for continuous ratings
Kappa handles categories: this rater said yes, that one said no. The intraclass correlation handles continuous ratings — pain scores, essay marks, joint-angle measurements — where raters differ by degree rather than by category.
The logic is a variance decomposition. If the subjects being rated genuinely differ from one another, most of the variation in the data sits between subjects. If raters disagree wildly, variation sits within subjects instead. The ICC is the first quantity as a proportion of the total, so it rises as rater noise falls.
Single versus average
Two figures come out of the same table and they answer different questions. ICC(1,1) is the reliability of one rater acting alone — the number to report if a single assessor will be used in practice. ICC(1,k) is the reliability of the mean across all k raters, which is always higher because averaging cancels individual error. Reporting the average-rater figure when the study design uses one rater overstates reliability substantially.
MSB is the between-subjects mean square and MSW the within-subjects mean square, both from a one-way ANOVA on the ratings. k is the number of raters per subject.
Worked example: MSB = 12.4, MSW = 2.1, k = 3
The gap between the mean squares is what reliability is made of:
- 1 Run a one-way ANOVA on the ratings. Subjects are the grouping factor. Read the between-subjects and within-subjects mean squares from the output.
- 2 Take the difference. 12.4 − 2.1 = 10.3. This is the part of the variance attributable to real differences between subjects.
- 3 Build the single-rater denominator. 12.4 + (3 − 1) × 2.1 = 12.4 + 4.2 = 16.6.
- 4 Divide for ICC(1,1). 10.3 ÷ 16.6 = .62 — moderate reliability for one rater.
- 5 Divide by MSB for ICC(1,k). 10.3 ÷ 12.4 = .83 — good reliability for the mean of three raters.
- 6 Report the one that matches your design. If your study uses a single assessor, .62 is the honest figure.
Koo & Li (2016) interpretation bands
Widely used benchmarks for reliability studies. Clinical measurement often demands the upper bands before a tool is considered usable.
| ICC | Reliability |
|---|---|
| Below .50 | Poor |
| .50 – .75 | Moderate |
| .75 – .90 | Good |
| Above .90 | Excellent |
Which ICC form to use
This calculator implements the one-way random-effects forms, ICC(1,1) and ICC(1,k). They fit designs where each subject is rated by a different set of raters drawn from a larger pool — the same situation Fleiss’ kappa covers for categories.
When the same raters assess every subject, a two-way model is appropriate instead, and the choice branches again: a two-way random model if those raters represent a wider population, or a two-way mixed model if they are the only raters of interest. Those forms need a three-way mean-square breakdown including a rater term, which a one-way ANOVA does not produce.
There is also a choice between absolute agreement and consistency. Consistency forgives a rater who is systematically two points high, as long as the ranking is preserved; absolute agreement does not. For measurement instruments the absolute form is usually what matters, and the distinction should be stated explicitly when reporting.