The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Open accessAbstract
This paper considers some appropriate and inappropriate uses of coefficient kappa and alternative kappa-like statistics. Discussion is restricted to the descriptive characteristics of these statistics for measuring agreement with categorical data in studies of reliability and validity. Special consideration is given to assumptions about whether marginals are fixed a priori, or free to vary. In reliability studies, when marginals are fixed, coefficient kappa is found to be appropriate. When either or both of the marginals are free to vary, however, it is suggested that the "chance" term in kappa be replaced by 1/ n, where n is the number of categories. In validity studies, we suggest considering whether one wants an index of improvement beyond "chance" or beyond the best a priori strategy employing base rates. In the former case, considerations are similar to those in reliability studies with the marginals for the criterion measure considered as fixed. In the latter case, it is suggested that the largest marginal proportion for the criterion measure be used in place of the "chance" term in kappa. Similarities and differences among these statistics are discussed and illustrated with synthetic data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Cited by
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
AUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179
Measuring intercultural sensitivity: The intercultural development inventory
Hammer, M. R., Bennett, M. J., & Wiseman, R. (2003) · International Journal of Intercultural Relations · doi:10.1016/s0147-1767(03)00032-4 · Cited by 1,732
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575
Multitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958
Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x04266540 · Cited by 829
Guidelines for establishing reliability when coding narrative data
Syed, M., & Nelson, S. C. (2015) · Emerging Adulthood · doi:10.1177/2167696815587648 · Cited by 801
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567
The SEGUE Framework for teaching and assessing communication skills
Makoul, G. (2001) · Patient Education and Counseling · doi:10.1016/s0738-3991(01)00136-7 · Cited by 457
Why Cohen’s Kappa should be avoided as performance measure in classification
Delgado, R., & Tibau, X.-A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222916 · Cited by 324
Meta-analysis of Cohen’s kappa
Sun, S. (2011) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-011-0077-3 · Cited by 282
Using Pooled Kappa to Summarize Interrater Agreement across Many Items
De Vries, H., Elliott, M. N., Kanouse, D. E., & Teleki, S. S. (2008) · Field Methods · doi:10.1177/1525822x08317166 · Cited by 254
The economic burden of visual impairment and blindness: a systematic review
Köberlein, J., Beifus, K., Schaffert, C., & Finger, R. P. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003471 · Cited by 226
Philippi, P., Baumeister, H., Apolinário-Hagen, J., Ebert, D. D., Hennemann, S., Kott, L., Lin, J., Messner, E.-M., & Terhorst, Y. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100459 · Cited by 200
“Full-texts of the remaining studies were obtained and screened for eligibility (PP, YT; agreement: 100%, κ n = 1) (Brennan and Prediger, 1981).”
BackgroundThe Effects of Right-Wing Populist Communication on Emotions and Cognitions toward Immigrants
Wirz, D. S., Wettstein, M., Schulz, A., Müller, P., Schemer, C., Ernst, N., Esser, F., & Wirth, W. (2018) · The International Journal of Press/Politics · doi:10.1177/1940161218788956 · Cited by 145
Bechdolf, A., Nelson, B., Cotton, S. M., Chanen, A., Thompson, A., Kettle, J., Conus, P., Amminger, G. P., Yung, A. R., Berk, M., et al. (2010) · Journal of Affective Disorders · doi:10.1016/j.jad.2010.06.016 · Cited by 131
Aickin, M. (1990) · Biometrics · doi:10.2307/2531434 · Cited by 122
Mahon, C., & Hevey, D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.626763 · Cited by 119
Assumptions behind Intercoder Reliability Indices
Zhao, X., Liu, J. S., & Deng, K. (2013) · Annals of the International Communication Association · doi:10.1080/23808985.2013.11679142 · Cited by 113
Ismail, A. I., Brodeur, J., Gagnon, P., Payette, M., Picard, D., Hamalian, T., Olivier, M., & Eastwood, B. J. (1992) · Community Dentistry and Oral Epidemiology · doi:10.1111/j.1600-0528.1992.tb01693.x · Cited by 99
Polnaszek, B., Gilmore-Bykovskyi, A., Hovanes, M., Roiland, R., Ferguson, P., Brown, R., & Kind, A. J. H. (2016) · Medical Care · doi:10.1097/mlr.0000000000000108 · Cited by 63
Digital game-based language learning for vocabulary development
Chowdhury, M., Dixon, L., Kuo, L.-J., Donaldson, J. P., Eslami, Z., Viruru, R., & Luo, W. (2024) · Computers and Education Open · doi:10.1016/j.caeo.2024.100160 · Cited by 64
A systematic review of peer assessment design elements
Alqassab, M., Strijbos, J.-W., Panadero, E., Ruiz, J. F., Warrens, M., & To, J. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09723-7 · Cited by 65
Large-Scale Observational Evidence of Cross-Cultural Differences in Facial Behavior
McDuff, D., Girard, J. M., & Kaliouby, R. E. (2017) · Journal of Nonverbal Behavior · doi:10.1007/s10919-016-0244-x · Cited by 60
Shapiro, A. F., Nahm, E. Y., Gottman, J. M., & Content, K. (2011) · American Journal of Orthopsychiatry · doi:10.1111/j.1939-0025.2011.01102.x · Cited by 55
The luck of the referee draw: the effect of exchanging reviews
BORNMANN, L., & DANIEL, H. (2009) · Learned Publishing · doi:10.1087/2009207 · Cited by 51
Mindfulness groups for early psychosis: A qualitative study
Ashcroft, K., Barrow, F., Lee, R., & MacKinnon, K. (2012) · Psychology and Psychotherapy Theory Research and Practice · doi:10.1111/j.2044-8341.2011.02031.x · Cited by 45
Olofsdotter, S., Sonnby, K., Vadlin, S., Furmark, T., & Nilsson, K. W. (2016) · Assessment · doi:10.1177/1073191115583858 · Cited by 41
THE EXACT VARIANCE OF WEIGHTED KAPPA WITH MULTIPLE RATERS
Mielke, P. W., Berry, K. J., & Johnston, J. E. (2007) · Psychological Reports · doi:10.2466/pr0.101.2.655-660 · Cited by 40
Generating Robot/Agent backchannels during a storytelling experiment
Al Moubayed, S., Baklouti, M., Chetouani, M., Dutoit, T., Mahdhaoui, A., Martin, J.-C., Ondas, S., Pelachaud, C., Urbain, J., & Yilmaz, M. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152572 · Cited by 33
Bosch, M., & Arntz, A. (2023) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2021.08.001 · Cited by 24
Developing and evaluating the extended epistemic vigilance framework
Bielik, T., & Krell, M. (2025) · Journal of Research in Science Teaching · doi:10.1002/tea.21983 · Cited by 16
Hubert's multi‐rater kappa revisited
Martín Andrés, A., & Álvarez Hernández, M. (2020) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12167 · Cited by 16
Christensen-Salem, A., Kinicki, A., Perrmann-Graham, J., & Walumbwa, F. O. (2023) · Human Relations · doi:10.1177/00187267221119767 · Cited by 16
A new coefficient of interrater agreement: The challenge of highly unequal category proportions.
van Oest, R. (2019) · Psychological Methods · doi:10.1037/met0000183 · Cited by 14
Hutmacher, F., Schläger, L., & Meerson, R. (2023) · Applied Cognitive Psychology · doi:10.1002/acp.4033 · Cited by 14
Autism and epistemology III: child development, behavioral stability, and reliability of measurement
Fisch, G. S. (2012) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.35269 · Cited by 13
Freund, J., Buntrock, C., Braun, L., Thielecke, J., Baumeister, H., Berking, M., Ebert, D. D., & Titzler, I. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100566 · Cited by 11
Braun, L., Freund, J., Thielecke, J., Baumeister, H., Ebert, D. D., & Titzler, I. (2022) · JMIR Mental Health · doi:10.2196/39122 · Cited by 11
Measuring Agreement Using Guessing Models and Knowledge Coefficients
Moss, J. (2023) · Psychometrika · doi:10.1007/s11336-023-09919-4 · Cited by 8
Using AI chatbots to facilitate mathematics preservice teachers’ noticing skills
Bastian, A., Buchholtz, N., & Kaiser, G. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1605921 · Cited by 7
Gerlach, C., Baus, M., Gianicolo, E., Bayer, O., Haugen, D. F., Weber, M., Mayland, C. R., ERANet-LAC CODE Core scientific group, Sigurdardottir, K., Hansen, M. I. T., et al. (2023) · Supportive Care in Cancer · doi:10.1007/s00520-022-07490-9 · Cited by 6
van Oest, R., & Girard, J. M. (2021) · Psychological Methods · doi:10.1037/met0000412 · Cited by 4
Decristan, J., Schastak, M., Reitenbach, V., & Rauch, D. (2021) · Zeitschrift für Erziehungswissenschaft · doi:10.1007/s11618-021-01023-6 · Cited by 4
Reitenbach, V., Decristan, J., Rauch, D. P., Bertram, V., & Schneider, K. M. (2023) · Unterrichtswissenschaft · doi:10.1007/s42010-023-00171-4 · Cited by 3
Quadratically Weighted Agreement Coefficients: Interpretations and Connections
van Oest, R., & Moss, J. (2026) · Psychometrika · doi:10.1017/psy.2026.10085
Metadata
- Authors
- Brennan, R. L., & Prediger, D. J.
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 1981-10
- Volume
- 41
- Issue
- 3
- Pages
- 687-699
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- OpenAlex
- W2149490731
- Cited by
- 1,570
- References
- 1
- FWCI
- 2.58
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
