Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 1,001–1,098 of 1,098
Interrater and Intrarater Reliability of the Beighton Score: A Systematic Review
Bockhorn, L. N., Vera, A. M., Dong, D., Delgado, D. A., Varner, K. E., & Harris, J. D. (2021) · Orthopaedic Journal of Sports Medicine · doi:10.1177/2325967120968099 · Cited by 85
ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics
Taleb, A., Kirchler, M., Monti, R., & Lippert, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02024 · Cited by 83
“The metric used in the task, as in the official Kaggle challenge, is the Quadratic Weighted Kappa (QwKappa [27]), which measures the agreement between two ratings.”
MethodDel Giacco, S. R., Cappai, A., Gambula, L., Cabras, S., Perra, S., Manconi, P. E., Carpiniello, B., & Pinna, F. (2016) · Respiratory Medicine · doi:10.1016/j.rmed.2016.09.014 · Cited by 85
Interpreting maps of science using citation context sentiments: a preliminary investigation
Small, H. (2011) · Scientometrics · doi:10.1007/s11192-011-0349-2 · Cited by 82
Women (Still) Ask For Less: Gender Differences in Hourly Rate in an Online Labor Marketplace
Foong, E., Vincent, N., Hecht, B., & Gerber, E. M. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274322 · Cited by 80
Playing to Wait: A Taxonomy of Idle Games
Alharthi, S. A., Alsaedi, O., Toups Dugas, P. O., Tanenbaum, T. J., & Hammer, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174195 · Cited by 79
“We used weighted kappa statistics to evaluate the agreement between the researchers on interactivity level and progress rate of the games [14].”
Castro, A., Gili, M., Ricci-Cabello, I., Roca, M., Gilbody, S., Perez-Ara, M. Á., Seguí, A., & McMillan, D. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.09.023 · Cited by 78
The disagreeable behaviour of the kappa statistic: L. Flight and S. A. Julious
Flight, L., & Julious, S. A. (2015) · Pharmaceutical Statistics · doi:10.1002/pst.1659 · Cited by 77
“Cohen’s kappa (Equation 1), however, assumes the disagreement between different categories is equally weighted, and the ordering of categories is not important [6].”
Transforming education with generative AI (GAI): key insights and future prospects
Lang, Q., Wang, M., Yin, M., Liang, S., & Song, W. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3537618 · Cited by 76
“To verify the scoring ability of LLMs, we can conduct a consistency check using the quadratic weighted Kappa coeffi-cient [53] based on the scoring results from baseline models such as convolutional neural networks (CNNs) and long short-term memory networks where S i,j represents the number of…”
Leaner leviathans: body condition variation in a critically endangered whale population
Bradford, A. L., Weller, D. W., Punt, A. E., Ivashchenko, Y. V., Burdin, A. M., VanBlaricom, G. R., & Brownell, R. L. (2012) · Journal of Mammalogy · doi:10.1644/11-mamm-a-091.1 · Cited by 76
“Given that we scored the postcranial condition using a 3-point ordinal scale, we employed a weighted kappa (kw) coefficient (Cohen 1968) with linear weighting in this case.”
MethodThe early effects of external and internal strategies on working memory updating training
Laine, M., Fellman, D., Waris, O., & Nyman, T. J. (2018) · Scientific Reports · doi:10.1038/s41598-018-22396-5 · Cited by 71
The impact of professional development in family child care: a practice-based approach
Koh, S., & Neuman, S. B. (2009) · Early Education and Development · doi:10.1080/10409280902908841 · Cited by 74
Physical Housing Environment: Development of a Self-Assessment Instrument
Fänge, A., & Iwarsson, S. (1999) · Canadian Journal of Occupational Therapy · doi:10.1177/000841749906600507 · Cited by 74
“Like kappa, weighted kappa is fully chance-corrected (Cohen, 1968).”
BackgroundMethodIdentity development in the late twenties: A never ending story.
Carlsson, J., Wängqvist, M., & Frisén, A. (2015) · Developmental Psychology · doi:10.1037/a0038745 · Cited by 73
Cash, A. H., Hamre, B. K., Pianta, R. C., & Myers, S. S. (2012) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2011.12.006 · Cited by 72
Reliability of Identification of Behavior Change Techniques in Intervention Descriptions
Abraham, C., Wood, C. E., Johnston, M., Francis, J., Hardeman, W., Richardson, M., & Michie, S. (2015) · Annals of Behavioral Medicine · doi:10.1007/s12160-015-9727-y · Cited by 71
Evidence-based Fact-Checking of Health-related Claims
Sarrouti, M., Ben Abacha, A., Mrabet, Y., & Demner-Fushman, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.297 · Cited by 71
“76 (Cohen, 1968), which indicates that the inter-annotator reliability is satisfactory, as the obtained k of 0.”
BackgroundBaccini, A., & De Nicolao, G. (2016) · Scientometrics · doi:10.1007/s11192-016-1929-y · Cited by 68
“When more than two nominal categories are considered, and when these categories are hierarchically ordered, weighted kappa can be introduced for describing specific patterns of disagreement (Cohen 1968).”
BackgroundAutomatic cell type identification methods for single-cell RNA sequencing
Xie, B., Jiang, Q., Mora, A., & Li, X. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.10.027 · Cited by 66
“The cell types assigned by the automatic methods can be given to cells or clusters. shown that for CellFishing.jl, when the features selected by scmap-cell were used, the Cohen’s kappa scores increased [63].”
MethodUncertainty Assessment of Hyperspectral Image Classification: Deep Learning vs. Random Forest
Shadman Roodposhti, M., Aryal, J., Lucieer, A., & Bryan, B. A. (2019) · Entropy · doi:10.3390/e21010078 · Cited by 65
Monsen, J., Odland, T., Faugli, A., Daae, E., & Eilertsen, D. (1995) · Psychotherapy Research · doi:10.1080/10503309512331331126 · Cited by 65
MR diagnosis of diaphragmatic endometriosis
Rousset, P., Gregory, J., Rousset-Jablonski, C., Hugon-Rodin, J., Regnard, J.-F., Chapron, C., Coste, J., Golfier, F., & Revel, M.-P. (2016) · European Radiology · doi:10.1007/s00330-016-4226-5 · Cited by 64
Antoine, J.-Y., Villaneau, J., & Lefeuvre, A. (2014) · Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1058 · Cited by 87
“(Davies and Fleiss, 1982) has proposed a generalization to any number of coders, while (Cohen, 1968) has defined a weighted version of the κ measure that fulfils better the need of reliability estimation for ordinal annotations: the disagreement between two ordinal annotations is no more binary, but depends on a Eucli…”
BackgroundEfficiency and safety of speech recognition for documentation in the electronic health record
Hodgson, T., Magrabi, F., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx073 · Cited by 63
“Inter-rater agreement was calculated using Cohen’s kappa.(11) Initial observed agreement was robust across all three categories: (1) 81.”
MethodSenkowski, D., Ziegler, T., Singh, M., Heinz, A., He, J., Silk, T., & Lorenz, R. C. (2024) · Neuropsychology Review · doi:10.1007/s11065-023-09592-5 · Cited by 59
“Cohen’s weighted Kappa was calculated for each individual domain (Cohen, 1968; Hallgren, 2012).”
MethodAutomation of Language Sample Analysis
Liu, H., MacWhinney, B., Fromm, D., & Lanzi, A. (2023) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2023_jslhr-22-00642 · Cited by 59
“Measures such as Cohen ’ s kappa (Cohen, 1968) cannot be used to determine reliability between two human transcribers because words, unlike codes, are not nominal variables.”
BackgroundHypothetical Vignettes in Empirical Bioethics Research
Ulrich, C. M., & Ratcliffe, S. J. (2007) · Advances in Bioethics · doi:10.1016/s1479-3709(07)11008-6 · Cited by 56
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55
Wilkinson‐Tough, M., Bocci, L., Thorne, K., & Herlihy, J. (2010) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.665 · Cited by 55
Measuring Patient Mobility in the ICU Using a Novel Noninvasive Sensor
Ma, A. J., Rawat, N., Reiter, A., Shrock, C., Zhan, A., Stone, A., Rabiee, A., Griffin, S., Needham, D. M., & Saria, S. (2017) · Critical Care Medicine · doi:10.1097/ccm.0000000000002265 · Cited by 53
A method for assessing fidelity of delivery of telephone behavioral support for smoking cessation
Lorencatto, F., West, R., Bruguera, C., & Michie, S. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0035149 · Cited by 52
“Given the high number of BCTs in the taxonomy (i.e. 45), the probability of chance selecting a particular code is low, and, as Kappa corrects for chance agreement amongst multiple coders, use of Kappa is likely to underestimate reliability (Steinijans, Diletti, Bomches, Greis, & Sikkeder, 1997).”
Influential(1993) · American Journal of Psychiatry · doi:10.1176/ajp.150.6.947 · Cited by 52
Peer Review versus Bibliometrics: Which Method Better Predicts the Scholarly Impact of Publications?
Abramo, G., D’Angelo, C. A., & Reale, E. (2019) · Scientometrics · doi:10.1007/s11192-019-03184-y · Cited by 50
“Since more than two ordered nominal categories are in place (the four VTR classes E, G, A, L), we use the weighted kappa, kw, (Cohen, 1968), where the weights indicate the seriousness of the disagreement.”
MethodModel Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science
Cambo, S. A., & Gergle, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501998 · Cited by 49
“This can typically involve validating the annotator choices through inter-annotator agreement measures like Krippendorff’s alpha or Cohen’s Kappa [24, 54].”
MethodInadequacies in the SES–Achievement model: Evidence from PISA and other studies
Marks, G. N., & O'Connell, M. (2021) · Review of Education · doi:10.1002/rev3.3293 · Cited by 47
“It is superior to simple percentages because it takes into account the possibility of agreement occurring by chance (Cohen, 1968).”
BackgroundLorencatto, F., West, R., Seymour, N., & Michie, S. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0032106 · Cited by 45
“The adapted framework was then piloted in waves 2 and 3 and reliability re-assessed in a cyclical and iterative process until high reliability (at least 75% agreement; Cohen, 1968) was achieved.”
MethodWalking and Non‐HDL‐C in Adults: A Meta‐Analysis of Randomized Controlled Trials
Kelley, G. A., Kelley, K. S., & Tran, Z. V. (2005) · Preventive Cardiology · doi:10.1111/j.1520-037x.2005.3474.x · Cited by 46
Methods for assessing accuracy and reliability in functional MRI
Le, T. H., & Hu, X. P. (1997) · NMR in Biomedicine · doi:10.1002/(sici)1099-1492(199706/08)10:4/5<160::aid-nbm458>3.0.co;2-a · Cited by 49
Risk factors for abdominal scar endometriosis after obstetric hysterotomies: a case-control study
DE OLIVEIRA, M. A. P., DE LEON, A. C. P., COUTINHO FREIRE, E., DE OLIVEIRA, H. C., & study, S. O. T. (2007) · Acta Obstetricia et Gynecologica Scandinavica · doi:10.1080/00016340601099346 · Cited by 45
“Quadratic weighted Kappa was used in the ordinal categorical variables with more than two levels (16).”
MethodTowards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach
Bayram, F., & Ahmed, B. S. (2025) · ACM Computing Surveys · doi:10.1145/3708497 · Cited by 48
“The EU requirements for trustworthy AI with different levels of abstraction are summarized in Figure 3.”
van Wijk, M. P., Breugem, C. C., & Kon, M. (2012) · Journal of Plastic, Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2011.08.021 · Cited by 45
Observations of Hyperactive Behaviour During Vigilance
Alberts, E., & van der Meere, J. (1992) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1992.tb00955.x · Cited by 44
Israel, B. A., Lachance, L., Coombe, C. M., Lee, S.-Y. D., Jensen, M., Wilson-Powers, E., Mentz, G., Muhammad, M., Rowe, Z., Reyes, A. G., et al. (2020) · Progress in Community Health Partnerships: Research, Education, and Action · doi:10.1353/cpr.2020.0015 · Cited by 45
“69,70 Following the conventional approach, CBPR partnership success would be indicated by averaging the scale scores of questionnaire items from all the members in the same partnership.”
Death fear in dying heart and cancer patients
Feifel, H., Freilich, J., & Hermann, L. J. (1973) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(73)90019-6 · Cited by 44
A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs
Toutanova, K., Brockett, C., Tran, K. M., & Amershi, S. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1033 · Cited by 42
Generating Product Descriptions from User Reviews
Novgorodov, S., Guy, I., Elad, G., & Radinsky, K. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313532 · Cited by 41
“The inter-annotator agreement for the main quality evaluation question, measured using weighted Kappa [10], was 0 .”
MethodKumar, A. P., Nayak, A., K, M. S., Chaitanya, & Ghosh, K. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00333-6 · Cited by 41
“Cohen’s Kappa coefficients gives the inter-rater agreement scores across the evaluation done between the two evaluators for each metric (Cohen, 1968).”
Observer Variation in Interpreting 18F-FDG PET/CT Findings for Lymphoma Staging
Hofman, M. S., Smeeton, N. C., Rankin, S. C., Nunan, T., & O'Doherty, M. J. (2009) · Journal of Nuclear Medicine · doi:10.2967/jnumed.109.064121 · Cited by 41
BRUGHA, T. S., NIENHUIS, F., BAGCHI, D., SMITH, J., & MELTZER, H. (1999) · Psychological Medicine · doi:10.1017/s003329179900834x · Cited by 41
“We examined the degree of concordance between clinical and lay interviews by applying a coefficient of agreement, Cohen’s kappa (Cohen, 1968).”
MethodTHE EXACT VARIANCE OF WEIGHTED KAPPA WITH MULTIPLE RATERS
Mielke, P. W., Berry, K. J., & Johnston, J. E. (2007) · Psychological Reports · doi:10.2466/pr0.101.2.655-660 · Cited by 40
The Reliability of Clinical Diagnoses: State of the Art
Kraemer, H. C. (2014) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032813-153739 · Cited by 39
“Cohen also introduced other kappa coefficients, including the weighted kappas k(w) (w between 0 and 1) (Berry et al. 2005, Cohen 1968, Fleiss & Cicchetti 1978, Fleiss et al. 1969), where the weight w is determined by the relative importance of the two types of errors (false positive and false…”
BackgroundSousa, L., de Mello, R., Cedrim, D., Garcia, A., Missier, P., Uchôa, A., Oliveira, A., & Romanovsky, A. (2018) · Information Systems · doi:10.1016/j.is.2018.02.003 · Cited by 39
Item Response Theory Investigation of Misophonia Auditory Triggers
Vitoratou, S., Uglik-Marucha, N., Hayes, C., Erfanian, M., Pearson, O., & Gregory, J. (2021) · Audiology Research · doi:10.3390/audiolres11040051 · Cited by 36
“Stability (test-retest reliability assessed at two weeks) was evaluated via Cohen’s [35] weighted Kappa (kw) for each item, following Landis and Koch [36] interpretations, along with the percentage of agreement and the Psi coefficient [37].”
MethodInter-observer reliability of DSM-5 substance use disorders
Denis, C. M., Gelernter, J., Hart, A. B., & Kranzler, H. R. (2015) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2015.05.019 · Cited by 36
Kearney, G. D., & McKenzie, S. (1993) · Cognitive Science · doi:10.1207/s15516709cog1704_5 · Cited by 36
“The kappa statistic (Cohen, 1960, 1968) was used to test for agreement among the five raters on the results in Table 7 with the following results: x = .467, &r(x) = .0013,2 = 13.13.”
MethodInfluentialInterpersonal process and outcome in variants of cognitive–behavioral psychotherapy
Critchfield, K. L., Henry, W. P., Castonguay, L. G., & Borkovec, T. D. (2007) · Journal of Clinical Psychology · doi:10.1002/jclp.20329 · Cited by 36
Wang, J., Redmond, S. J., Bertoux, M., Hodges, J. R., & Hornberger, M. (2016) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2016.00119 · Cited by 35
Asymptotic variability of (multilevel) multirater kappa coefficients
Vanbelle, S. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218794733 · Cited by 33
“A total of 30 subjects were classified as suffering mainly of (1) depression, (2) personality disorder, (3) schizophrenia, (4) neurosis or (5) other psychiatric disorder.”
BackgroundComparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials
Armijo‐Olivo, S., Craig, R., & Campbell, S. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1398 · Cited by 31
Warmth and contingency and their relationship to maternal attitudes toward parenting
Keller, H., Lohaus, A., Völker, S., Elben, C., & Ball, J. (2003) · The Journal of Genetic Psychology · doi:10.1080/00221320309597984 · Cited by 32
Store-and-Forward Images in Teledermatology: Narrative Literature Review
Jiang, S. W., Flynn, M. S., Kwock, J. T., & Nicholas, M. W. (2022) · JMIR Dermatology · doi:10.2196/37517 · Cited by 32
“When responses pertain to the diagnosis of a disease, it is called diagnostic agreement or concordance, which can be reported as exact agreement between evaluators or as the sum of exact agreement and weighted partial agreement of categorically similar diagnoses [11].”
BackgroundBholla, M., Kapalata, N., Masika, E., Chande, H., Jugheli, L., Sasamalo, M., Glass, T. R., Beck, H.-P., & Reither, K. (2016) · BMC Infectious Diseases · doi:10.1186/s12879-016-1578-z · Cited by 31
“Sensitivity, specificity and Cohen’s Kappa [32] were calculated for each and for all diagnostic modalities combined with culture and cytology as single reference standards, and with culture and cytology combined (M.”
MethodA Simple Guide to Inter-rater, Intra-rater and Test-retest Reliability for Animal Behaviour Studies
Harvey, N. D. (2021) · doi:10.31219/osf.io/8stpy · Cited by 31
Gerdle, G. L. (1999) · Scandinavian Journal of Rehabilitation Medicine · doi:10.1080/003655099444362 · Cited by 30
Towards Evaluating Narrative Quality In Student Writing
Somasundaran, S., Flor, M., Chodorow, M., Molloy, H., Gyawali, B., & McCulla, L. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00007 · Cited by 29
Motivating and Shaping Scientific Argumentation in Lab Reports
Gouvea, J., Appleby, L., Fu, L., & Wagh, A. (2022) · CBE—Life Sciences Education · doi:10.1187/cbe.21-11-0316 · Cited by 27
“Because the SOLO categories are ordinal, we calculated interrater reliability of initial preconsensus coding of the full data set using Cronbach’s kappa with linear weighting (Cohen, 1968).”
MethodEffect of pharmacotherapy for anxiety disorders on quality of life: a meta-analysis
Hofmann, S. G., Wu, J. Q., Boettcher, H., & Sturm, J. (2014) · Quality of Life Research · doi:10.1007/s11136-013-0573-8 · Cited by 24
“Validity assessment The following modified Jadad criteria [13] were used by three of the authors (QJW, HB, and JS) and three independent trained assessors to determine study quality: (1) the study was described as randomized; (2) participants were adequately randomized; (3) the study was described as double blind; (4)…”
MethodKoegel, L. K., Navab, A., Ashbaugh, K., & Koegel, R. L. (2016) · Journal of Positive Behavior Interventions · doi:10.1177/1098300715596136 · Cited by 23
“Because this measure takes into account agreement by chance but allows for incorporation of scaled degrees of disagreement, it is considered a robust measure of interobserver agreement for categorical data (Cohen, 1968, 1988).”
MethodKrippendorff, K. (2016) · Methodology · doi:10.1027/1614-2241/a000119 · Cited by 22
“To the best of my knowledge, the weighted κ (Cohen, 1968) and Lin’s (1989) ρr are not defined for rank ordered data, as Feng claims, at least not by customary conceptions of rank order differences; for example, in Spearman’s rank order correlation coefficient ρ (rho).”
MethodRot, C., Creutzinger, K., Goetz, H., Winder, C., Morrison, J., Conboy, M., Bajus, A., & Renaud, D. L. (2022) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2022.105630 · Cited by 21
Agreement Among Different Scales for Causality Assessment in Drug-Induced Liver Injury
Das, S., Behera, S. K., Xavier, A. S., Velupula, S., Dkhar, S. A., & Selvarajan, S. (2018) · Clinical Drug Investigation · doi:10.1007/s40261-017-0601-5 · Cited by 19
Corrected Zegers-ten Berge Coefficients Are Special Cases of Cohen’s Weighted Kappa
Warrens, M. J. (2014) · Journal of Classification · doi:10.1007/s00357-014-9156-9 · Cited by 18
“Connections between Pearson’s r and weighted kappa have been found by Cohen (1960, 1968) and Schuster (2004).”
BackgroundMethodInfluentialClarke, E., Hull, L., Loomes, R., McCormick, C. E. B., Sheinkopf, S. J., & Mandy, W. (2021) · Research in Autism Spectrum Disorders · doi:10.1016/j.rasd.2021.101844 · Cited by 17
“Inter-rater reliability was judged by calculating the Cohen’s ĸ coefficient for each item (Cohen, 1968).”
MethodHubert's multi‐rater kappa revisited
Martín Andrés, A., & Álvarez Hernández, M. (2020) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12167 · Cited by 16
Effectiveness of Peer Assessment in a Professionalism Course Using an Online Workshop
David Strang, K. (2015) · Journal of Information Technology Education: Innovations in Practice · doi:10.28945/2100 · Cited by 16
“Pairwise assessment consistency can be estimated by Kappa interrater reliability (Cohen, 1968). However, there were no best-practices for evaluating the reliability of multiple assessments– beyond two students although the literature contained several relevant studies. Zhang and Blakey (2012) used factor analysis for…”
BackgroundMethodInfluentialVidotto, G., Anselmi, P., Filipponi, L., Tommasi, M., & Saggino, A. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01100 · Cited by 16
“Cohen’s k (Cohen, 1968) was computed on all the 24!”
MethodKumar, N., Kaushik, R., Zhang, K. Y. J., Uversky, V. N., Sahu, U., Sood, R., & Bhatia, S. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26467 · Cited by 15
“SARS-CoV-2 spike protein to the ACE2 receptor has direct implications not just for identifying the host range of SARS-CoV-2,(45) but also for adaptation and infection dynamics in the host.(40) It is worth noting that the potent NAbs have a high affinity for”
BackgroundMethod(2016) · Contemporary issues in technology and teacher education · Cited by 15
Autism and epistemology III: child development, behavioral stability, and reliability of measurement
Fisch, G. S. (2012) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.35269 · Cited by 13
Strategies and cues adolescents use to assess the age of an online stranger
Groenestein, E., Baas, N., van Deursen, A. J. A. M., & de Jong, M. D. T. (2018) · Information, Communication & Society · doi:10.1080/1369118x.2017.1309443 · Cited by 13
“After one round and revision, the second round of coding of 15% of the data provided a Cohen’s Kappa of .80 which sufficed for continuation (Cohen, 1968).”
MethodDiagnosis of placental malaria in poorly fixed and processed placental tissue
Liu, Y., Griffin, J. B., Muehlenbachs, A., Rogerson, S. J., Bailis, A. J., Sharma, R., Sullivan, D. J., Tshefu, A. K., Landis, S. H., Kabongo, J.-M. M., et al. (2016) · Malaria Journal · doi:10.1186/s12936-016-1314-6 · Cited by 9
(2022) · doi:10.1101/2022.07.09.22277429 · Cited by 9
Burley, J., & McAloon, J. (2024) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2023.11.003 · Cited by 13
Automated Progress-Monitoring for Literate Language Use in Narrative Assessment (LLUNA)
Fox, C., Jones, S., Gillam, S. L., Israelsen-Augenstein, M., Schwartz, S., & Gillam, R. B. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.894478 · Cited by 8
“K qw has several advantages over other classification metrics in that it can both weight the probability of chance agreement between raters and differentially weight the distance of disagreement between raters (Cohen, 1968).”
BackgroundMeasuring Agreement Using Guessing Models and Knowledge Coefficients
Moss, J. (2023) · Psychometrika · doi:10.1007/s11336-023-09919-4 · Cited by 8
Inter‐population differences in acetabular senescence: relevance in age‐at‐death estimation
San-Millán, M., & Rissech, C. (2023) · International Journal of Legal Medicine · doi:10.1007/s00414-023-02954-x · Cited by 8
Konrad, J., Loos, S., Neumann, P., Zentner, N., Mayer, B., Slade, M., Jordan, H., De Rosa, C., Del Vecchio, V., Égerházi, A., et al. (2015) · Journal of Mental Health · doi:10.3109/09638237.2014.951478 · Cited by 7
“…the meeting, weighted kappa (kw) was used, which is a modification of the kappa coefficient allowing for differential weighting of disagreement involving three matrices: the matrix of observed scores, the matrix of expected scores based on chance agreement and the weight matrix (Cohen, 1968).”
MethodMeasuring mindfulness: A systematic review and evaluation of self-report mindfulness scales
Chems-Maarif, R., Cavanagh, K., Baer, R., Gu, J., & Strauss, C. (2026) · Clinical Psychology Review · doi:10.1016/j.cpr.2026.102719 · Cited by 7
Guidelines, training and quality assurance: influence on general practitioner MRI referral quality
Kara, S., Smart, A., Officer, T., Dassanayake, C., Clark, P., Smit, A., & Cavadino, A. (2019) · The Journal of Primary Health Care · doi:10.1071/hc19034 · Cited by 6
Li, S. N., Chien, W. T., Lam, S. K. K., Chong, Y. Y., & Gloster, A. T. (2026) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-024-06610-0 · Cited by 5
“The weighted kappa statistic for the Psy-Flex-C’s 2-week retesting interval was calculated to explore its stability (Cohen, 1968), and a result of ≥ 0.70 indicated acceptable stability (Kline, 2000).”
InfluentialAI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training
Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C. (2026) · Archives of Disease in Childhood · doi:10.1136/archdischild-2025-329846 · Cited by 4
“⇒ Disparities observed between (human) patient and clinician assessment for the GPT agents suggest that diverse viewpoints are needed to fully capture the experiential learning associated with clinical communication.”
van Oest, R., & Girard, J. M. (2021) · Psychological Methods · doi:10.1037/met0000412 · Cited by 4
Abedin, I., Putra, A., Kang, H.-E., Singh, A., Singh, S., Jung, W.-K., Kim, H.-W., & Kundu, S. (2025) · doi:10.21203/rs.3.rs-7140563/v1 · Cited by 1
Quadratically Weighted Agreement Coefficients: Interpretations and Connections
van Oest, R., & Moss, J. (2026) · Psychometrika · doi:10.1017/psy.2026.10085
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“Following previous work with a similar annotation scheme (Wang et al., 2024a), we use quadratic-weighted Kappa ( κ 2 ) (Cohen, 1968), an extension of Cohen’s Kappa that accounts for the magnitude of disagreements, making it suitable for ordinal data.”
A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
Chen, J., Chen, W., Lin, Z., & Vong, C. M. (2026) · arXiv · doi:10.48550/arxiv.2608.24419
“Judge-versus-human agreement is chance-corrected [8]; where the verdict space is ordered we also report the linearly weighted variant [9], because an ordered space punishes an adjacent-class error and an opposite-class error identically under the unweighted statistic, and the two are not equally…”
Using the ADI-R to diagnose autism in preschool children
Lord, C., Storoschuk, S., Rutter, M., & Pickles, A. (1993) · Infant Mental Health Journal · doi:10.1002/1097-0355(199323)14:3<234::aid-imhj2280140308>3.0.co;2-f
Rosario (1988) · doi:10.1002/1520-6629(198801)16:1<55::aid-jcop2290160108>3.0.co;2-u
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.