Beyond kappa: A review of interrater agreement measures
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
In 1960, Cohen introduced the kappa coefficient to measure chance‐corrected nominal scale agreement between two raters. Since then, numerous extensions and generalizations of this interrater agreement measure have been proposed in the literature. This paper reviews and critiques various approaches to the study of interrater agreement, for which the relevant data comprise either nominal or ordinal categorical ratings from multiple raters. It presents a comprehensive compilation of the main statistical approaches to this problem, descriptions and characterizations of the underlying models, and discussions of related statistical methodologies for estimation and confidence‐interval construction. The emphasis is on various practical scenarios and designs that underlie the development of these measures, and the interrelationships between them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Overall the inter reviewer concordance was Kappa = 0.85, suggesting an excellent agreement (Banerjee, Capozzoli, McSweeney, & Sinha, 1999).”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Banerjee, Capozzoli, McSweeney, & Sinha, 1999
In the citing paper
“A Cohen's Kappa .75 and above is considered excellent agreement beyond chance (Banerjee et al., 1999).”
From Critical Race Theory (CRT) and colourism: a manifestation of whitewashing in marketing communications? (2020) · cites this paper as Banerjee et al., 1999
In the citing paper
“Still, one existing result is that Fleiss' kappa converges to the one-factor intraclass correlation coefficient, where the items constitute the only factor due to interchangeable raters (Banerjee et al., 1999;Janson & Olsson, 2004;Landis & Koch, 1977a;Vanbelle, 2019).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Banerjee et al., 1999
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Fleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. H. (1979) · Springer Series in Statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,244
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Distributions in Statistics: Continuous Multivariate Distributions.
Gokhale, D. V., Johnson, N. L., & Kotz, S. (1973) · Biometrics · doi:10.2307/2529696 · Cited by 1,599
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Statistical Methods for Rates and Proportions.
D., F. N., & Fleiss, J. L. (1973) · Biometrics · doi:10.2307/2529186 · Cited by 1,388
Simple Models for the Analysis of Association in Cross-Classifications Having Ordered Categories
Goodman, L. A. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10481650 · Cited by 889
Misinterpretation and misuse of the kappa statistic
MACLURE, M., & WILLETT, W. C. (1987) · American Journal of Epidemiology · doi:10.1093/aje/126.2.161 · Cited by 808
Statistical Methods for Rates and Proportions.
Fielding, A., & Fleiss, J. L. (1974) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344565 · Cited by 767
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
Measures of Association for Cross Classifications*
Goodman, L. A., & Kruskal, W. H. (1954) · Journal of the American Statistical Association · doi:10.1080/01621459.1954.10501231 · Cited by 678
A One-Way Components of Variance Model for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529465 · Cited by 478
Pearson, K. (1900) · Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character · doi:10.1098/rsta.1900.0022 · Cited by 476
Measuring Agreement for Multinomial Data
Davies, M., & Fleiss, J. L. (1982) · Biometrics · doi:10.2307/2529886 · Cited by 353
Review papers : Design and analysis of reliability studies
Dunn, G. (1992) · Statistical Methods in Medical Research · doi:10.1177/096228029200100202 · Cited by 325
2 x 2 Kappa Coefficients: Measures of Agreement or Association
Bloch, D. A., & Kraemer, H. C. (1989) · Biometrics · doi:10.2307/2532052 · Cited by 291
Donner, A., & Eliasziw, M. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111109 · Cited by 266
Extension of the Kappa Coefficient
Kraemer, H. C. (1980) · Biometrics · doi:10.2307/2529972 · Cited by 237
Measuring interrater reliability among multiple raters: An example of methods for nominal data
Posner, K. L., Sampson, P. D., Caplan, R. A., Ward, R. J., & Cheney, F. W. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090917 · Cited by 229
Another look at interrater agreement.
Zwick, R. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.3.374 · Cited by 224
Comparison of the Null Distributions of Weighted Kappa and the C Ordinal Statistic
Cicchetti, D. V., & Fleiss, J. L. (1977) · Applied Psychological Measurement · doi:10.1177/014662167700100206 · Cited by 204
Modelling patterns of agreement and disagreement
Agresti, A. (1992) · Statistical Methods in Medical Research · doi:10.1177/096228029200100205 · Cited by 165
Modeling Agreement among Raters
Tanner, M. A., & Young, M. A. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477157 · Cited by 161
A Model for Agreement Between Ratings on an Ordinal Scale
Agresti, A. (1988) · Biometrics · doi:10.2307/2531866 · Cited by 158
Design and Analysis of Reliability Studies
Gleser, L. J., & Dunn, G. (1991) · Technometrics · doi:10.2307/1269420 · Cited by 157
MOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 153
Inference About Weighted Kappa in the Non-Null Case
Fleiss, J. L., & Cicchetti, D. V. (1978) · Applied Psychological Measurement · doi:10.1177/014662167800200111 · Cited by 140
Latent class analysis of diagnostic agreement
Uebersax, J. S., & Grove, W. M. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090509 · Cited by 125
Aickin, M. (1990) · Biometrics · doi:10.2307/2531434 · Cited by 122
The Maximum Likelihood Estimation of Correlation from Contingency Tables
Tallis, G. M. (1962) · Biometrics · doi:10.2307/2527476 · Cited by 121
Distribution and Heritability of Recurrent Ear Infections
Kværner, K. J., Harris, J. R., Tambs, K., et al. (1997) · Annals of Otology Rhinology & Laryngology · doi:10.1177/000348949710600802 · Cited by 114
The statistical analysis of kappa statistics in multiple samples
Donner, A. P., & Klar, N. S. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(96)00057-1 · Cited by 94
A Weighted Concordance Correlation Coefficient for Repeated Measurement Designs
Chinchilli, V. M., Martel, J. K., Kumanyika, S., & Lloyd, T. (1996) · Biometrics · doi:10.2307/2533172 · Cited by 92
General Observer-Agreement Measures on Individual Subjects and Groups of Subjects
O’Connell, D. L., & Dobson, A. J. (1984) · Biometrics · doi:10.2307/2531148 · Cited by 81
Testing the Homogeneity of Kappa Statistics
Donner, A. P., Eliasziw, M., & Klar, N. S. (1996) · Biometrics · doi:10.2307/2533154 · Cited by 74
Haberman, S. J. (1988) · Sociological Methodology · doi:10.2307/271049 · Cited by 72
Quasi-Symmetric Latent Class Models, with Application to Rater Agreement
Agresti, A., & Lang, J. B. (1993) · Biometrics · doi:10.2307/2532608 · Cited by 66
A comparison of methods for calculating a stratified kappa
Barlow, W. E., Lai, M., & Azen, S. P. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100913 · Cited by 61
Familial influences on the clinical characteristics of major depression: a twin study
Kendler, K. S., Neale, M. C., Kessler, R. C., Heath, A. C., & Eaves, L. J. (1992) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1992.tb03283.x · Cited by 59
Kappa muddles together two sources of disagreement: Tetrachoric correlation is preferable
Hutchinson, T. P. (1993) · Research in Nursing & Health · doi:10.1002/nur.4770160410 · Cited by 59
Latent Variable Models for Clustered Ordinal Data
Qu, Y., Piedmonte, M. R., & MEDENDORP, S. V. (1995) · Biometrics · doi:10.2307/2533332 · Cited by 59
JACKKNIFING FUNCTIONS OF MULTINOMIAL FREQUENCIES, WITH AN APPLICATION TO A MEASURE OF CONCORDANCE1
Fleiss, J. L., & DAVIES, M. (1982) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a113371 · Cited by 55
Latent variable models for ordered categorical data
Bartholomew, D. J. (1983) · Journal of Econometrics · doi:10.1016/0304-4076(83)90101-x · Cited by 53
Modeling ordinal scale disagreement.
Tanner, M. A., & Young, M. A. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.98.2.408 · Cited by 51
The epidemiology of pregnancy complications and outcome in a Norwegian twin population.
Corey, L. A., Berg, K., Solaas, M. H., et al. (1992) · PubMed · Cited by 45
Latent Variable Models for Clustered Dichotomous Data with Multiple Subclusters
Qu, Y., Williams, G. W., Beck, G. J., et al. (1992) · Biometrics · doi:10.2307/2532701 · Cited by 42
Measurement of Interrater Agreement with Adjustment for Covariates
Barlow, W. E. (1996) · Biometrics · doi:10.2307/2532907 · Cited by 40
The Equivalence of Tetrachoric and Maximum Likelihood Estimates of ρ in 2 × 2 Tables
Hamdan, M. A. (1970) · Biometrika · doi:10.1093/biomet/57.1.212 · Cited by 36
How many raters? toward the most reliable diagnostic consensus
KRAEMER, H. C. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110305 · Cited by 35
Estimating kappa from binocular data
Oden, N. L. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100813 · Cited by 31
Kraemer, H. C. (1997) · Archives of General Psychiatry · doi:10.1001/archpsyc.1997.01830240081011 · Cited by 31
Modelling covariate effects in observer agreement studies: The case of nominal scale agreement
Graham, P. J. (1995) · Statistics in Medicine · doi:10.1002/sim.4780140308 · Cited by 29
Assessing Interrater Agreement from Dependent Data
Williamson, J. M., & Manatunga, A. K. (1997) · Biometrics · doi:10.2307/2533970 · Cited by 28
Estimating kappa from binocular data and comparing marginal probabilities
Schouten, H. J. A. (1993) · Statistics in Medicine · doi:10.1002/sim.4780122306 · Cited by 26
Maximum likelihood estimation of the kappa coefficient from models of matched binary responses
Shoukri, M. M., Martin, S. W., & Mian, I. U. H. (1995) · Statistics in Medicine · doi:10.1002/sim.4780140109 · Cited by 23
A HIERARCHICAL APPROACH TO INFERENCES CONCERNING INTEROBSERVER AGREEMENT FOR MULTINOMIAL DATA
Donner, A. P., & Eliasziw, M. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19970530)16:10<1081::aid-sim521>3.0.co;2-x · Cited by 15
Irwig, L. M., & GROENEVELD, H. T. (1988) · Statistics in Medicine · doi:10.1002/sim.4780070907 · Cited by 7
Research works printed but not linked: 3
- 10.1002/(SICI)1097-0258(19970530)16:10<1097::AID-SIM523>3.0.CO;2-8 · doi:10.1002/(sici)1097-0258(19970530)16:10<1097::aid-sim523>3.0.co;2-8
- 10.2307/2281536 · doi:10.2307/2281536 · Paper page
- Snedecor G. W., 1967, Statistical Methods
Cited by
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
“The statistical foundations of these measures are beyond the scope of this article but are fully discussed elsewhere (Banerjee et al., 1999; Davey et al., 2010; Feng, 2013; Hallgren, 2012; Hayes & Krippendorff, 2007; Rust & Cooil, 1994).”
MethodCritical thinking, cognitive presence, and computer conferencing in distance education
Garrison, D. R., Anderson, T., & Archer, W. (2001) · American Journal of Distance Education · doi:10.1080/08923640109527071 · Cited by 3,044
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
The malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899
Metadata
- Authors
- Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D.
- Affiliations
- Wayne State University (US)
- University of New Hampshire (US)
- Venue
- Canadian Journal of Statistics
- Publisher
- Wiley
- Published
- 1999-03
- Volume
- 27
- Issue
- 1
- Pages
- 3-23
- ISSN
- 0319-5724 (ISSN-L)
- 1708-945X
- OpenAlex
- W2076898331
- Cited by
- 976
- References
- 66
- Citations by year
- 2026: 25 · 2025: 39 · 2024: 50 · 2023: 46 · 2022: 40 · 2021: 67 · 2020: 59 · 2019: 59 · 2018: 49 · 2017: 55 · 2016: 53 · 2015: 61 · 2014: 64 · 2013: 54 · 2012: 38
- FWCI
- 9.17
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Canadian Journal of Statistics (publishedVersion)
- DigitalCommons - Fairfield (Fairfield University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- interobserver agreement, kappa statistic, chance-corrected agreement, multiple raters, confidence intervals, categorical data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.