Fleiss’ Kappa
Chance-corrected agreement across three or more raters and any number of categories.
10 subjects, 4 raters, 2 categories. Mean observed agreement 0.717, chance agreement 0.511.
Fleiss’ kappa extends chance-corrected agreement to any number of raters. For 10 subjects rated by 4 raters into 2 categories, with mean observed agreement of .717 and chance agreement of .511, kappa is (.717 − .511) ÷ (1 − .511) = .42 — moderate agreement.
Agreement when there are more than two raters
Cohen’s kappa compares two specific raters. Fleiss’ kappa answers a different question: given a fixed number of ratings per subject, how much do raters agree overall? Crucially, it does not require the same raters to assess every subject — only the same number of them — which suits designs where coders are drawn from a pool.
The calculation works subject by subject. For each one it asks how many rater pairs agreed out of all possible pairs, then averages that across subjects to get observed agreement. Chance agreement comes from how often each category was chosen overall, squared and summed. The final step is the familiar kappa ratio.
The input format
Each line is one subject, and the numbers on that line are how many raters chose each category. A line reading 3, 1 means three raters chose the first category and one chose the second. Every line must sum to the same number of raters, which is what makes the calculation valid.
n is the number of subjects, N the number of raters per subject, and nᵢⱼ how many raters assigned subject i to category j.
Worked example: 10 subjects, 4 raters, 2 categories
Agreement within subjects first, then the chance baseline:
- 1 Lay out the ratings table. One row per subject; each cell holds how many of the 4 raters chose that category. Every row totals 4.
- 2 Score agreement within each subject. For a row of 3 and 1: (3² + 1² − 4) ÷ (4 × 3) = 6 ÷ 12 = .50. A unanimous row of 4 and 0 gives (16 − 4) ÷ 12 = 1.00.
- 3 Average across subjects. The mean of the ten row values gives P̄ = .717.
- 4 Find the category proportions. Category 1 received 23 of the 40 ratings and category 2 received 17, so p₁ = .575 and p₂ = .425.
- 5 Compute chance agreement. .575² + .425² = .3306 + .1806 = .511.
- 6 Apply the kappa formula. (.717 − .511) ÷ (1 − .511) = .206 ÷ .489 = .42, which is moderate agreement.
Landis & Koch (1977) interpretation bands
The same benchmarks used for Cohen’s kappa. They are widely cited conventions rather than statistical thresholds.
| Kappa | Strength of agreement |
|---|---|
| Below .00 | Poor — worse than chance |
| .00 – .20 | Slight |
| .21 – .40 | Fair |
| .41 – .60 | Moderate |
| .61 – .80 | Substantial |
| .81 – 1.00 | Almost perfect |
What Fleiss’ kappa does not tell you
The coefficient is a single number for the whole coding exercise. It cannot say which rater is out of step, nor which category causes the trouble — and in practice disagreement is rarely spread evenly. Inspecting the per-subject agreement values, and the category proportions, usually points at the problem faster than the kappa itself.
Like Cohen’s kappa, it is sensitive to how the categories are distributed: when one category dominates, chance agreement rises and kappa falls even though raters appear to agree. Reporting the mean observed agreement alongside kappa lets a reader see this.
Fleiss’ kappa also treats every disagreement as equal, so it is not appropriate for ordered categories where some disagreements are worse than others. For ordinal ratings, a weighted coefficient or the intraclass correlation is the better fit.