The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Beyond kappa: A review of interrater agreement measures
Abstract
In 1960, Cohen introduced the kappa coefficient to measure chance‐corrected nominal scale agreement between two raters. Since then, numerous extensions and generalizations of this interrater agreement measure have been proposed in the literature. This paper reviews and critiques various approaches to the study of interrater agreement, for which the relevant data comprise either nominal or ordinal categorical ratings from multiple raters. It presents a comprehensive compilation of the main statistical approaches to this problem, descriptions and characterizations of the underlying models, and discussions of related statistical methodologies for estimation and confidence‐interval construction. The emphasis is on various practical scenarios and designs that underlie the development of these measures, and the interrelationships between them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Overall the inter reviewer concordance was Kappa = 0.85, suggesting an excellent agreement (Banerjee, Capozzoli, McSweeney, & Sinha, 1999).”
From Acceptance and commitment therapy - Do we know enough? Cumulative and sequential meta-analyses of randomized controlled trials. (2016) · cites this paper as Banerjee, Capozzoli, McSweeney, & Sinha, 1999
In the citing paper
“A Cohen's Kappa .75 and above is considered excellent agreement beyond chance (Banerjee et al., 1999).”
From Critical Race Theory (CRT) and colourism: a manifestation of whitewashing in marketing communications? (2020) · cites this paper as Banerjee et al., 1999
In the citing paper
“Still, one existing result is that Fleiss' kappa converges to the one-factor intraclass correlation coefficient, where the items constitute the only factor due to interchangeable raters (Banerjee et al., 1999;Janson & Olsson, 2004;Landis & Koch, 1977a;Vanbelle, 2019).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Banerjee et al., 1999
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Fleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034
Measures of Association for Cross Classifications
Goodman, L. A., & Kruskal, W. H. (1979) · Springer Series in Statistics · doi:10.1007/978-1-4612-9995-0 · Cited by 2,244
Research works printed but not linked: 3
- 10.1002/(SICI)1097-0258(19970530)16:10<1097::AID-SIM523>3.0.CO;2-8 · doi:10.1002/(sici)1097-0258(19970530)16:10<1097::aid-sim523>3.0.co;2-8
- 10.2307/2281536 · doi:10.2307/2281536 · Paper page
- Snedecor G. W., 1967, Statistical Methods
Cited by
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
“The statistical foundations of these measures are beyond the scope of this article but are fully discussed elsewhere (Banerjee et al., 1999; Davey et al., 2010; Feng, 2013; Hallgren, 2012; Hayes & Krippendorff, 2007; Rust & Cooil, 1994).”
MethodCritical thinking, cognitive presence, and computer conferencing in distance education
Garrison, D. R., Anderson, T., & Archer, W. (2001) · American Journal of Distance Education · doi:10.1080/08923640109527071 · Cited by 3,044
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
The malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899
ASSESSING TEACHING PRESENCE IN A COMPUTER CONFERENCING CONTEXT
Anderson, T., Rourke, L., Garrison, R., & Archer, W. (2001) · Online Learning · doi:10.24059/olj.v5i2.1875 · Cited by 1,885
(2013) · Cited by 1,351
Method agreement analysis: A review of correct methodology
Watson, P. F., & Petrie, A. (2010) · Theriogenology · doi:10.1016/j.theriogenology.2010.01.003 · Cited by 881
Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x04266540 · Cited by 829
Are more resources always better for growth? Resource stickiness in market and product expansion
Mishina, Y., Pollock, T. G., & Porac, J. F. (2004) · Strategic Management Journal · doi:10.1002/smj.424 · Cited by 674
Foody, G. M. (2020) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.111630 · Cited by 665
Kappa coefficients in medical research
Chmura Kraemer, H., Periyakoil, V. S., & Noda, A. (2002) · Statistics in Medicine · doi:10.1002/sim.1180 · Cited by 527
Introductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295779 · Cited by 485
Chicco, D., Warrens, M. J., & Jurman, G. (2021) · IEEE Access · doi:10.1109/access.2021.3084050 · Cited by 444
College Students and Sexual Consent: Unique Insights
Jozkowski, K. N., & Peterson, Z. D. (2013) · Journal of Sex Research · doi:10.1080/00224499.2012.700739 · Cited by 377
Snapping, pinning, liking or texting: Investigating social media in higher education beyond Facebook
Manca, S. (2020) · The Internet and Higher Education · doi:10.1016/j.iheduc.2019.100707 · Cited by 347
Jozkowski, K. N., Peterson, Z. D., Sanders, S. A., Dennis, B., & Reece, M. (2014) · The Journal of Sex Research · doi:10.1080/00224499.2013.792326 · Cited by 322
Fostering knowledge construction in university students through asynchronous discussion groups
Schellens, T., & Valcke, M. (2006) · Computers & Education · doi:10.1016/j.compedu.2004.07.010 · Cited by 301
Meta-analysis of Cohen’s kappa
Sun, S. (2011) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-011-0077-3 · Cited by 282
Graf, S., Paolini, S., & Rubin, M. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2052 · Cited by 286
Content Analysis—A Methodological Primer for Gender Research
Neuendorf, K. A. (2011) · Sex Roles · doi:10.1007/s11199-010-9893-0 · Cited by 272
An Examination of Questionnaire Evaluation by Expert Reviewers
Olson, K. (2010) · Field Methods · doi:10.1177/1525822x10379795 · Cited by 253
An Examination of the Use of Large Language Models to Aid Analysis of Textual Data
(2024) · doi:10.1177/16094069241231168 · Cited by 227
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207
Rith-Najarian, L. R., Boustani, M. M., & Chorpita, B. F. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.06.035 · Cited by 176
Pauley, D., Cuijpers, P., Papola, D., Miguel, C., & Karyotaki, E. (2021) · Psychological Medicine · doi:10.1017/s0033291721001999 · Cited by 170
“The Cohen ’ s Kappa for overall risk of bias judgements was perfect at 1.0, while ICC assessing agreement on each individual domain judgement was good to excellent at 0.88 [95% CI: 0.81 – 0.92].”
InfluentialHacker, T., Stone, P., & MacBeth, A. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.10.053 · Cited by 130
Anti-Smoking Media Campaign Messages: Theory and Practice
Cohen, E. L., Shumate, M. D., & Gold, A. (2007) · Health Communication · doi:10.1080/10410230701453884 · Cited by 124
(2019) · doi:10.1145/3287324.3287374 · Cited by 104
Mitchell, T. A. (2020) · Journal of Marketing Management · doi:10.1080/0267257x.2020.1794934 · Cited by 62
On the credibility perception of news on Twitter: Readers, topics and features
Shariff, S. M., Zhang, X., & Sanderson, M. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.06.026 · Cited by 49
Chieu, V. M., Herbst, P., & Weiss, M. (2011) · Journal of the Learning Sciences · doi:10.1080/10508406.2011.528324 · Cited by 46
Henkel, O., Hills, L., Boxer, A., Roberts, B., & Levonian, Z. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3664693 · Cited by 42
An empirical study of unspecified dependencies in make-based build systems
Bezemer, C.-P., McIntosh, S., Adams, B., German, D. M., & Hassan, A. E. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9510-8 · Cited by 42
THE EXACT VARIANCE OF WEIGHTED KAPPA WITH MULTIPLE RATERS
Mielke, P. W., Berry, K. J., & Johnston, J. E. (2007) · Psychological Reports · doi:10.2466/pr0.101.2.655-660 · Cited by 40
(2008) · Cited by 36
Meyer, F., Meissel, K., & McNaughton, S. (2017) · Journal of Research in Reading · doi:10.1111/1467-9817.12061 · Cited by 30
A new coefficient of interrater agreement: The challenge of highly unequal category proportions.
van Oest, R. (2019) · Psychological Methods · doi:10.1037/met0000183 · Cited by 14
Clinical diagnosis of completeness of medical abortion by nurses: a reliability study in Mozambique
Gebreselassie, H., Ustá, M., Andersen, K. L., & Mitchell, E. M. H. (2012) · Contraception · doi:10.1016/j.contraception.2011.08.012 · Cited by 5
van Oest, R., & Girard, J. M. (2021) · Psychological Methods · doi:10.1037/met0000412 · Cited by 4
Quadratically Weighted Agreement Coefficients: Interpretations and Connections
van Oest, R., & Moss, J. (2026) · Psychometrika · doi:10.1017/psy.2026.10085
Metadata
- Authors
- Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D.
- Affiliations
- Wayne State University (US)
- University of New Hampshire (US)
- Venue
- Canadian Journal of Statistics
- Publisher
- Wiley
- Published
- 1999-03
- Volume
- 27
- Issue
- 1
- Pages
- 3-23
- ISSN
- 0319-5724 (ISSN-L)
- 1708-945X
- OpenAlex
- W2076898331
- Cited by
- 976
- References
- 66
- Citations by year
- 2026: 25 · 2025: 39 · 2024: 50 · 2023: 46 · 2022: 40 · 2021: 67 · 2020: 59 · 2019: 59 · 2018: 49 · 2017: 55 · 2016: 53 · 2015: 61 · 2014: 64 · 2013: 54 · 2012: 38
- FWCI
- 9.17
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Canadian Journal of Statistics (publishedVersion)
- DigitalCommons - Fairfield (Fairfield University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- interobserver agreement, kappa statistic, chance-corrected agreement, multiple raters, confidence intervals, categorical data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.