Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 201–300 of 1,098
Singh, A., Klarner, P., & Hess, T. (2020) · Long Range Planning · doi:10.1016/j.lrp.2019.07.001 · Cited by 298
Motivating Knowledge Sharing in Knowledge Management Systems: A Quasi–Field Experiment
Wang, S., Noe, R. A., & Wang, Z.-M. (2014) · Journal of Management · doi:10.1177/0149206311412192 · Cited by 290
“Based on Wasko and Faraj’s study, intercoder reliability was assessed for the overall ratings using weighted Cohen’s kappa (Cohen, 1968), which corrects for the expected level of chance correlation between raters and weights agreement by the extent of divergence between raters.”
MethodPreoperative MR Imaging of Anal Fistulas: Does It Really Help the Surgeon?
Beets-Tan, R. G. H., Beets, G. L., van der Hoop, A. G., Kessels, A. G. H., Vliegen, R. F. A., Baeten, C. G. M. I., & van Engelshoven, J. M. A. (2001) · Radiology · doi:10.1148/radiology.218.1.r01dc0575 · Cited by 338
Evaluating transdisciplinary science
Stokols, D., Fuqua, J., Gress, J., Harvey, R., Phillips, K., Baezconde-Garbanati, L., Unger, J., Palmer, P., Clark, M., Colby, S., et al. (2003) · Nicotine & Tobacco Research · doi:10.1080/14622200310001625555 · Cited by 328
“The average interrater reliability coefficient (Cohen’s kappa) for the form over the first 3 years of the TTURC grant is .94 (see Cohen, 1968; Krippendorff, 1980).”
BackgroundCalcium-channel blockade and incidence of cancer in aged populations
Pahor, M., Guralnik, J. M., Ferrucci, L., Corti, M.-C., Salive, M. E., Cerhan, J. R., Wallace, R. B., & Havlik, R. J. (1996) · The Lancet · doi:10.1016/s0140-6736(96)04277-8 · Cited by 340
Vitamin D levels in people with multiple sclerosis and community controls in Tasmania, Australia
Mei, I. A. F., Ponsonby, A.-L., Dwyer, T., Blizzard, L., Taylor, B. V., Kilpatrick, T., Butzkueven, H., & McMichael, A. J. (2007) · Journal of Neurology · doi:10.1007/s00415-006-0315-8 · Cited by 321
Social interaction skills and theory of mind in young children.
Watson, A. C., Nixon, C. L., Wilson, A., & Capage, L. (1999) · Developmental Psychology · doi:10.1037//0012-1649.35.2.386 · Cited by 313
Inductive Reasoning: A Training Approach
Klauer, K. J., & Phye, G. D. (2008) · Review of Educational Research · doi:10.3102/0034654307313402 · Cited by 301
“Since the early efforts of Cohen (1968) and Glass (1976), effect sizes have typically been estimated as standardized mean differences.”
BackgroundReview papers : Design and analysis of reliability studies
Dunn, G. (1992) · Statistical Methods in Medical Research · doi:10.1177/096228029200100202 · Cited by 325
The Importance of Quality of Primary Studies in Producing Unbiased Systematic Reviews
Khan, K. S. (1996) · Archives of Internal Medicine · doi:10.1001/archinte.156.6.661 · Cited by 328
An Overview on Assessing Agreement with Continuous Measurements
Barnhart, H. X., Haber, M. J., & Lin, L. I. (2007) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543400701376480 · Cited by 319
“…− Yi2k|δ|Yi1k′ − Yi2k′|δ] EI [ ∑p k=1 ∑p k′=1 dkk′|Yi1k − Yi2k|δ|Yi1k′ − Yi2k′|δ] , where it reduces to the repeated measures CCC if δ = 1 and to a parameter comparable to repeated measures version of kappa (Cohen, 1960, 1968) index for assessing agreement between categorical measurements if δ = 0.”
Dietary Choline Intake: Current State of Knowledge Across the Life Cycle
Wiedeman, A., Barr, S., Green, T., Xu, Z., Innis, S., & Kitts, D. (2018) · Nutrients · doi:10.3390/nu10101513 · Cited by 316
Evidence-based Diagnostics: Adult Septic Arthritis
Carpenter, C. R., Schuur, J. D., Everett, W. W., & Pines, J. M. (2011) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2011.01121.x · Cited by 314
Pollock, T. G., Rindova, V. P., & Maggitti, P. G. (2008) · Academy of Management Journal · doi:10.5465/amj.2008.31767275 · Cited by 323
“We calculated intercoder reliability using Cohen’s kappa ( Cohen, 1968 ), which captures the degree of agreement between two coders, adjusted for the likelihood of agreement by chance.”
MethodtDCS in patients with disorders of consciousness: Sham-controlled randomized double-blind study
Thibaut, A., Bruno, M.-A., Ledoux, D., Demertzi, A., & Laureys, S. (2014) · Neurology · doi:10.1212/wnl.0000000000000260 · Cited by 324
Assessing Inter-Rater Reliability for Rating Scales: Resolving some Basic Issues
Cicchetti, D. V. (1976) · British Journal of Psychiatry · doi:10.1192/bjp.129.5.452 · Cited by 321
Cleland, J. A., Childs, J. D., Fritz, J. M., Whitman, J. M., & Eberhart, S. L. (2007) · Physical Therapy · doi:10.2522/ptj.20060155 · Cited by 318
Velikonja, T., Fett, A.-K., & Velthorst, E. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.3645 · Cited by 292
Social impact and social preference as determinants of children's peer group status.
Newcomb, A. F., & Bukowski, W. M. (1983) · Developmental Psychology · doi:10.1037//0012-1649.19.6.856 · Cited by 310
Which is the Best Activity Rating Scale for Patients Undergoing Total Joint Arthroplasty?
Naal, F. D., Impellizzeri, F. M., & Leunig, M. (2009) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-008-0358-5 · Cited by 317
“The reliability of the three activity rating scales (UCLA, Tegner, ARS) was examined using the weighted Cohen’s kappa kw [10].”
MethodAssessing what to address in science communication
Bruine de Bruin, W., & Bostrom, A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212729110 · Cited by 290
Adverse reactions to antidepressants
Uher, R., Farmer, A., Henigsberg, N., Rietschel, M., Mors, O., Maier, W., Kozel, D., Hauser, J., Souery, D., Placentino, A., et al. (2009) · British Journal of Psychiatry · doi:10.1192/bjp.bp.108.061960 · Cited by 302
Role of autobiographical memory in social problem solving and depression.
Goddard, L., Dritschel, B., & Burton, A. (1996) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.105.4.609 · Cited by 304
Ballman, K. V., Buckner, J. C., Brown, P. D., Giannini, C., Flynn, P. J., LaPlant, B. R., & Jaeckle, K. A. (2007) · Neuro-Oncology · doi:10.1215/15228517-2006-025 · Cited by 314
“A kappa statistic (Cohen, 1968) and 95% CI were used to summarize the amount of agreement above and beyond that expected by chance alone.”
MethodMassage Reduces Anxiety in Child and Adolescent Psychiatric Patients
FIELD, T., MORROW, C., VALDEON, C., LARSON, S., KUHN, C., & SCHANBERG, S. (1992) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199201000-00019 · Cited by 310
Da Luz, L. T., Nascimento, B., Shankarakutty, A. K., Rizoli, S., & Adhikari, N. K. (2014) · Critical Care · doi:10.1186/s13054-014-0518-9 · Cited by 308
“Agreement between reviewers was assessed by using the Cohen κ [6].”
MethodMacDonald, S. L. S., Rubens, M. B., Hansell, D. M., Copley, S. J., Desai, S. R., du Bois, R. M., Nicholson, A. G., Colby, T. V., & Wells, A. U. (2001) · Radiology · doi:10.1148/radiol.2213010158 · Cited by 310
Measuring the impact of menopausal symptoms on quality of life.
Daly, E., Gray, A., Barlow, D., McPherson, K., Roche, M., & Vessey, M. (1993) · BMJ · doi:10.1136/bmj.307.6908.836 · Cited by 312
Boerma, E. C., Mathura, K. R., van der Voort, P. H., Spronk, P. E., & Ince, C. (2005) · Critical Care · doi:10.1186/cc3809 · Cited by 308
Coplan, P. M., Schmader, K., Nikas, A., Chan, I. S. F., Choo, P., Levin, M. J., Johnson, G., Bauer, M., Williams, H. M., Kaplan, K. M., et al. (2004) · The Journal of Pain · doi:10.1016/j.jpain.2004.06.001 · Cited by 306
Diagnosing Tests: Using and Misusing Diagnostic and Screening Tests
Streiner, D. L. (2003) · Journal of Personality Assessment · doi:10.1207/s15327752jpa8103_03 · Cited by 300
“7This is a somewhat unfortunate choice of terms, as Cohen (1968) previously used the same one, which he abbreviated as κw, to refer to κ scaled for partial agreement.”
MethodWind power generation: A review and a research agenda
Vargas, S. A., Esteves, G. R. T., Maçaira, P. M., Bastos, B. Q., Cyrino Oliveira, F. L., & Souza, R. C. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.02.015 · Cited by 295
Marital quality and gender differences in parent^child interaction.
Kerig, P. K., Cowan, P. A., & Cowan, C. P. (1993) · Developmental Psychology · doi:10.1037//0012-1649.29.6.931 · Cited by 296
Halder, B., Bandyopadhyay, J., & Banik, P. (2021) · Sustainable Cities and Society · doi:10.1016/j.scs.2021.103186 · Cited by 292
Behavioral Signal Processing: Deriving Human Behavioral Informatics From Speech and Language
Narayanan, S., & Georgiou, P. G. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2236291 · Cited by 294
“Interannotator reliability can be measured in many different ways, e.g., using percentage agreement, Holsti's method [25], Scott's [26], Cohen's [27], Fleiss' f [28], and Krippendorff's [29].”
MethodCicchetti, D. V., Shoinralter, D., & Tyrer, P. J. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900103 · Cited by 291
“These findings are quite consistent with the recent caveats expressed by Cohen (1983), who quantified the substantial losses in information (accuracy) that can occur when a continuous scale of measurement is dichotomized.”
MethodResultCurley, M. A. Q., Harris, S. K., Fraser, K. A., Johnson, R. A., & Arnold, J. H. (2006) · Pediatric Critical Care Medicine · doi:10.1097/01.pcc.0000200955.40962.38 · Cited by 299
“[32] We examined interrater agreement on the NRS rating, a continuous variable, with intraclass correlation coefficients generated in SPSS 12.”
MethodCoping with job stress and burnout in the human services.
Shinn, M., Rosario, M., Mørch, H., & Chestnut, D. E. (1984) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.46.4.864 · Cited by 295
Advanced statistics: Understanding Medical Record Review (MRR) Studies
Worster, A., & Haines, T. (2004) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2004.tb01433.x · Cited by 269
“For such more complex, multivariate analyses, tests such as the weighted kappa are more suitable.(25) It is beyond the scope of this paper to discuss the different statistical tests for correlation analysis and agreement, but consideration needs to be given to factors such as whether the agreement is between observers…”
BackgroundThe International Fitness Scale (IFIS): usefulness of self-reported fitness in youth
Ortega, F. B., Ruiz, J. R., España-Romero, V., Vicente-Rodriguez, G., Martínez-Gómez, D., Manios, Y., Béghin, L., Molnar, D., Widhalm, K., Moreno, L. A., et al. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr039 · Cited by 267
Oxman, A. D., Muir, D. C. F., Shannon, H. S., Stock, S. R., Hnizdo, E., & Lange, H. J. (1993) · American Review of Respiratory Disease · doi:10.1164/ajrccm/148.1.38 · Cited by 303
Peripheral vascular tree stenoses: evaluation with moving-bed infusion-tracking MR angiography.
Ho, K. Y., Leiner, T., de Haan, M. W., Kessels, A. G., Kitslaar, P. J., & van Engelshoven, J. M. (1998) · Radiology · doi:10.1148/radiology.206.3.9494486 · Cited by 301
“The linear, weighted K statistic (5,6) was used to assess interobserver agree-”
MethodReview of existing grading systems for cervical or lumbar disc and facet joint degeneration
Kettler, A., & Wilke, H.-J. (2006) · European Spine Journal · doi:10.1007/s00586-005-0954-y · Cited by 294
“To assess the reliability either Kappa [12] or Intraclass Correlation Coefficients (ICC) [57] have to be determined.”
BackgroundMethodInfluentialIterative fully convolutional neural networks for automatic vertebra segmentation and identification
Lessmann, N., van Ginneken, B., de Jong, P. A., & Išgum, I. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.02.005 · Cited by 266
Move to the Beat—Rhythms of Change and Firm Performance
Klarner, P., & Raisch, S. (2013) · Academy of Management Journal · doi:10.5465/amj.2010.0767 · Cited by 277
Biederman, J., Faraone, S. V., & Monuteaux, M. C. (2002) · American Journal of Psychiatry · doi:10.1176/appi.ajp.159.9.1556 · Cited by 290
Dynamic Resistance Training as Stand‐Alone Antihypertensive Lifestyle Therapy: A Meta‐Analysis
MacDonald, H. V., Johnson, B. T., Huedo‐Medina, T. B., Livingston, J., Forsyth, K. C., Kraemer, W. J., Farinatti, P. T. V., & Pescatello, L. S. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.116.003231 · Cited by 273
Expressive control during a disappointment: Variations related to preschoolers' behavior problems.
Cole, P. M., Zahn-Waxler, C., & Smith, K. D. (1994) · Developmental Psychology · doi:10.1037/0012-1649.30.6.835 · Cited by 292
Klabunde, C. N., Vernon, S. W., Nadel, M. R., Breen, N., Seeff, L. C., & Brown, M. L. (2005) · Medical Care · doi:10.1097/01.mlr.0000173599.67470.ba · Cited by 287
Chen, W., Xie, X., Peng, J., Shahabi, H., Hong, H., Bui, D. T., Duan, Z., Li, S., & Zhu, A.-X. (2018) · CATENA · doi:10.1016/j.catena.2018.01.012 · Cited by 281
2 x 2 Kappa Coefficients: Measures of Agreement or Association
Bloch, D. A., & Kraemer, H. C. (1989) · Biometrics · doi:10.2307/2532052 · Cited by 291
The active movement scale: An evaluative tool for infants with obstetrical brachial plexus palsy
Curtis, C., Stephens, D., Clarke, H. M., & Andrews, D. (2002) · The Journal of Hand Surgery · doi:10.1053/jhsu.2002.32965 · Cited by 293
de Bree, E., Koops, W., Kröger, R., van Ruth, S., Witkamp, A. J., & Zoetmulder, F. A. N. (2004) · Journal of Surgical Oncology · doi:10.1002/jso.20049 · Cited by 291
“The agreement between intraoperative and CT-findings as well as the interobserver agreement were assessed using the Cohen’s Kappa (k) index of agreement [12] and with the McNemar chisquare test.”
Hicks, G. E., Fritz, J. M., Delitto, A., & Mishock, J. (2003) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(03)00365-4 · Cited by 285
Gupta, J. K., Chien, P. F. W., Voit, D., Clark, T. J., & Khan, K. S. (2002) · Acta Obstetricia et Gynecologica Scandinavica · doi:10.1034/j.1600-0412.2001.810902.x · Cited by 282
Fluet-Chouinard, E., Lehner, B., Rebelo, L.-M., Papa, F., & Hamilton, S. K. (2015) · Remote Sensing of Environment · doi:10.1016/j.rse.2014.10.015 · Cited by 274
Duits, L. C., Phoa, K. N., Curvers, W. L., ten Kate, F. J. W., Meijer, G. A., Seldenrijk, C. A., Offerhaus, G. J., Visser, M., Meijer, S. L., Krishnadath, K. K., et al. (2015) · Gut · doi:10.1136/gutjnl-2014-307278 · Cited by 286
Which treatment for whom for ADHD? Moderators of treatment response in the MTA.
Owens, E. B., Hinshaw, S. P., Kraemer, H. C., Arnold, L. E., Abikoff, H. B., Cantwell, D. P., Conners, C. K., Elliott, G., Greenhill, L. L., Hechtman, L., et al. (2003) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.71.3.540 · Cited by 275
“When the two types of error are considered equally important, the appropriate measure is the unweighted kappa (Cohen, 1968; Fleiss, 1981).”
BackgroundPuerperal Psychosis: Phenomena and Diagnosis
Brockington, I. F. (1981) · Archives of General Psychiatry · doi:10.1001/archpsyc.1981.01780320109013 · Cited by 284
Stratification by Skin Color in Contemporary Mexico
Villarreal, A. (2010) · American Sociological Review · doi:10.1177/0003122410378232 · Cited by 274
Chien, P. F. W., Arnott, N., Gordon, A., Owen, P., & Khan, K. S. (2000) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2000.tb11690.x · Cited by 283
Brooks, J., Waylen, K. A., & Mulder, M. B. (2013) · Environmental Evidence · doi:10.1186/2047-2382-2-2 · Cited by 252
“Cohen’s kappa represents the proportion of agreement after accounting for the level of agreement expected by chance when coding categorical data (Cohen 1960) and Cohen’s weighted kappa for ordinal data [82,83].”
MethodMeta-analysis of Cohen’s kappa
Sun, S. (2011) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-011-0077-3 · Cited by 282
“In fact, Cohen’s j has been extended to estimate inter-rater reliability for many complicated rating scenarios (e.g., Berry and Mielke 1998; Cohen 1968, 1972; Davies and Fleiss 1982; Fleiss 1971; Gross 1986; Janson and Olsson 2001, 2004; Kraemer 1980; Kraemer et al. 2004; Landis and Koch 1977b;…”
BackgroundMcElwain, N. L., Halberstadt, A. G., & Volling, B. L. (2007) · Child Development · doi:10.1111/j.1467-8624.2007.01074.x · Cited by 262
“Cohen's weighted kappa was utilized to assess interobserver agreement, as it is considered a more appropriate type of reliability statistic for rating scales (Cohen, 1968).”
MethodMEADE, M. O., COOK, R. J., GUYATT, G. H., GROLL, R., KACHURA, J. R., BEDARD, M., COOK, D. J., SLUTSKY, A. S., & STEWART, T. E. (2000) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.161.1.9809003 · Cited by 280
Diabetic Retinopathy Detection Using Transfer Learning and Deep Learning
Gangwar, A. K., & Ravi, V. (2021) · Advances in Intelligent Systems and Computing · doi:10.1007/978-981-15-5788-0_64 · Cited by 245
Cladribine in treatment of chronic progressive multiple sclerosis
SIPE, J. (1994) · The Lancet · doi:10.1016/s0140-6736(94)91046-4 · Cited by 283
Klales, A. R., Ousley, S. D., & Vollner, J. M. (2012) · American Journal of Physical Anthropology · doi:10.1002/ajpa.22102 · Cited by 277
Post, B., Merkus, M. P., de Bie, R. M. A., de Haan, R. J., & Speelman, J. D. (2005) · Movement Disorders · doi:10.1002/mds.20640 · Cited by 266
Cost-sensitive learning for imbalanced medical data: a review
Araf, I., Idri, A., & Chairi, I. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10652-8 · Cited by 209
Autism Spectrum Disorders in the Stockholm Youth Cohort: Design, Prevalence and Validity
Idring, S., Rai, D., Dal, H., Dalman, C., Sturm, H., Zander, E., Lee, B. K., Serlachius, E., & Magnusson, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041280 · Cited by 275
“Inter-rater reliability was calculated as a kappa (K) coefficient [23] of concordance of final case status (overall and by presence of comorbid intellectual disability) between the two reviewers.”
MethodInterexpert Agreement of Plus Disease Diagnosis in Retinopathy of Prematurity
Chiang, M. F. (2007) · Archives of Ophthalmology · doi:10.1001/archopht.125.7.875 · Cited by 275
Lamb, M. E., Thompson, R. A., Gardner, W. P., Charnov, E. L., & Estes, D. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00026522 · Cited by 266
Emotional Dynamics in the Age of Misinformation
Zollo, F., Novak, P. K., Del Vicario, M., Bessi, A., Mozetič, I., Scala, A., Caldarelli, G., & Quattrociocchi, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138740 · Cited by 273
“We use Cohen’s weighted kappa [50] to compare the inter-annotator agreement and self-agreement.”
MethodMethodological issues in measuring health-related quality of life
Lin, X.-J., Lin, I.-M., & Fan, S.-Y. (2013) · Tzu Chi Medical Journal · doi:10.1016/j.tcmj.2012.09.002 · Cited by 245
“Inter-rater reliability refers to the consistency of measures between different raters, which is normally assessed using the k statistic [46].”
MethodCLAUDETTE: an automated detector of potentially unfair clauses in online terms of service
Lippi, M., Pałka, P., Contissa, G., Lagioia, F., Micklitz, H.-W., Sartor, G., & Torroni, P. (2019) · Artificial Intelligence and Law · doi:10.1007/s10506-019-09243-2 · Cited by 169
“In order to quantitatively measure the inter-annotation agreement, for this test set we computed the standard Cohen’s metric (Cohen 1968), which resulted to be 0.871, a value that is typically considered as an “almost perfect agreement” (Landis and Koch 1977).”
MethodInfluentialEscape Rooms for Learning: A Systematic Review
(2019) · Proceedings of the 12th European Conference on Game Based Learning · doi:10.34190/gbl.19.179 · Cited by 166
“The interrater reliability (IRR) for the total scores was attained using Cohen’s kappa (Cohen, 1968); this value was 0.91, showing good agreement between the two authors.”
MethodInfluentialEvaluation of the Clinical Age-Related Maculopathy Staging System
SEDDON, J., SHARMA, S., & ADELMAN, R. (2006) · Ophthalmology · doi:10.1016/j.ophtha.2005.11.001 · Cited by 263
“Measures of reliability, including exact agreement, agreement within 1 grade, unweighted , and weighted , which allowed a 1-step difference, were calculated.(13,14) To calculate these s, a 1.”
MethodMaternal Responsiveness to Infants in Three Societies: The United States, France, and Japan
Bornstein, M. H., Tamis‐LeMonda, C. S., Tal, J., Ludemann, P. M., Toda, S., Rahn, C. W., Pêcheux, M., Azuma, H., & Vardi, D. (1992) · Child Development · doi:10.1111/j.1467-8624.1992.tb01663.x · Cited by 216
“Reliability of scoring was assessed using kappa (Cohen, 1968; Hartmann & Wood, 1982).”
MethodA Theoretical Approach to Web Design in E-Commerce: A Belief Reinforcement Model
Song, J., & Zahedi, F. “. (2005) · Management Science · doi:10.1287/mnsc.1050.0427 · Cited by 257
Systematic review of artificial intelligence in education: trends, benefits, and challenges
Garzón, J., Patiño, E., & Marulanda, C. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9080084 · Cited by 265
“This value was 0.89, indicating almost perfect agreement [39].”
InfluentialParsing Framing Processes: The Interplay Between Online Public Opinion and Media Coverage
Zhou, Y., & Moy, P. (2007) · Journal of Communication · doi:10.1111/j.0021-9916.2007.00330.x · Cited by 229
Assessment of soil quality indices in agricultural lands of Qazvin Province, Iran
Rahmanipour, F., Marzaioli, R., Bahrami, H. A., Fereidouni, Z., & Bandarabadi, S. R. (2014) · Ecological Indicators · doi:10.1016/j.ecolind.2013.12.003 · Cited by 259
Mehrholz, J., Wagner, K., Meißner, D., Grundmann, K., Zange, C., Koch, R., & Pohl, M. (2005) · Clinical Rehabilitation · doi:10.1191/0269215505cr889oa · Cited by 262
Kelley, G. A., & Kelley, K. S. (2009) · Preventive Medicine · doi:10.1016/j.ypmed.2008.10.010 · Cited by 263
“Using Cohen's kappa statistic (Cohen, 1968), the overall agreement rate prior to adjudication was 0.”
MethodGhaye, B., Ghuysen, A., Willems, V., Lambermont, B., Gerard, P., D'Orio, V., Gevenois, P. A., & Dondelinger, R. F. (2006) · Radiology · doi:10.1148/radiol.2392050075 · Cited by 263
“00), further calculations for qualitative variables were based on the results of the more experienced reader (reader 1) (32).”
MethodAn Instrument for Assessing the Organizational Benefits of IS Projects
Mirani, R., & Lederer, A. L. (1998) · Decision Sciences · doi:10.1111/j.1540-5915.1998.tb00878.x · Cited by 260
Kujan, O., Khattab, A., Oliver, R. J., Roberts, S. A., Thakker, N., & Sloan, P. (2007) · Oral Oncology · doi:10.1016/j.oraloncology.2006.03.009 · Cited by 258
Resilience through the lens of interactionism: a systematic review.
Pangallo, A., Zibarras, L., Lewis, R., & Flaxman, P. (2015) · Psychological Assessment · doi:10.1037/pas0000024 · Cited by 261
Edwards, J., Berube, M., Erlandson, K., Haug, S., Johnstone, H., Meagher, M., Sarkodee-Adoo, S., & Zwicker, J. G. (2011) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/dbp.0b013e31822a396a · Cited by 244
Automatic scoring of non-native spontaneous speech in tests of spoken English
Zechner, K., Higgins, D., Xi, X., & Williamson, D. M. (2009) · Speech Communication · doi:10.1016/j.specom.2009.04.009 · Cited by 249
“We use quadratically weighted Kappa (Cohen, 1968) throughout the paper, where the weights correspond to the squares of score differences.”
MethodStroke Severity Is a Crucial Predictor of Outcome: An International Prospective Validation Study
Rost, N. S., Bottle, A., Lee, J., Randall, M., Middleton, S., Shaw, L., Thijs, V., Rinkel, G. J. E., Hemmen, T. M., & collaborators, T. G. C. S. G. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.115.002433 · Cited by 247
Are dietary patterns stable throughout early and mid-childhood? A birth cohort study
Northstone, K., & Emmett, P. M. (2008) · British Journal of Nutrition · doi:10.1017/s0007114508968264 · Cited by 252
“Finally, component scores were split into quintiles and weighted k((21)) was used to compare scores between each pair of time-points.”
MethodExercise and sleep: a systematic review of previous meta‐analyses
Kelley, G. A., & Kelley, K. S. (2017) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12236 · Cited by 258
Hotel companies' environmental policies and practices: a content analysis of their web pages
Hsieh, Y. (. (2012) · International Journal of Contemporary Hospitality Management · doi:10.1108/095961112 · Cited by 237
van Dijk, M., Roofthooft, D. W. E., Anand, K. J. S., Guldemond, F., de Graaf, J., Simons, S., de Jager, Y., van Goudoever, J. B., & Tibboel, D. (2009) · The Clinical Journal of Pain · doi:10.1097/ajp.0b013e3181a5b52a · Cited by 250
Skeem, J. L., & Cauffman, E. (2003) · Behavioral Sciences & the Law · doi:10.1002/bsl.563 · Cited by 247
“10 Second, to assess chance-corrected levels of agreement at a more specific level, weighted (Cohen, 1968) was computed (using unit weights) for scores on each of the 20 items.”
MethodSex-related strategies for coping with interpersonal conflict in children aged five and seven.
Miller, P. M., Danaher, D. L., & Forbes, D. L. (1986) · Developmental Psychology · doi:10.1037//0012-1649.22.4.543 · Cited by 251
Dependence of Weighted Kappa Coefficients on the Number of Categories
Brenner, H., & Kliebsch, U. (1996) · Epidemiology · doi:10.1097/00001648-199603000-00016 · Cited by 255
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.