Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 101–200 of 1,098
Eating in the absence of hunger and overweight in girls from 5 to 7 y of age,,
Fisher, J. O., & Birch, L. L. (2002) · The American Journal of Clinical Nutrition · doi:10.1093/ajcn/76.1.226 · Cited by 596
“Cohen’s Kappa statistic was used to assess the stability or tracking of the girls’ eating in the absence of hunger within individuals over both time points (17).”
MethodVariability in Radiologists' Interpretations of Mammograms
Elmore, J. G., Wells, C. K., Lee, C. H., Howard, D. H., & Feinstein, A. R. (1994) · New England Journal of Medicine · doi:10.1056/nejm199412013312206 · Cited by 601
Marlovits, S., Singer, P., Zeller, P., Mandl, I., Haller, J., & Trattnig, S. (2006) · European Journal of Radiology · doi:10.1016/j.ejrad.2005.08.007 · Cited by 602
Blood vessel segmentation algorithms — Review of methods, datasets and evaluation metrics
Moccia, S., De Momi, E., El Hadji, S., & Mattos, L. S. (2018) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.02.001 · Cited by 567
Childhood sexual and physical abuse in adult patients with borderline personality disorder
(1990) · American Journal of Psychiatry · doi:10.1176/ajp.147.8.1008 · Cited by 571
Childhood Experience of Care and Abuse (CECA): A Retrospective Interview Measure
Bifulco, A., Brown, G. W., & Harris, T. O. (1994) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1994.tb01284.x · Cited by 579
Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts
Mollick, E., & Nanda, R. (2016) · Management Science · doi:10.1287/mnsc.2015.2207 · Cited by 515
“Using a squared weighting scheme appropriate for subjective scoring (Cohen, 1968), the kappa statistic which measures the inter‐rater agreement for these judges was 0.”
InfluentialDo AI chatbots improve students learning outcomes? Evidence from a meta‐analysis
Wu, R., & Yu, Z. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13334 · Cited by 576
Hillmayr, D., Ziernwald, L., Reinhold, F., Hofer, S. I., & Reiss, K. M. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103897 · Cited by 574
“Moreover, the low values of Cohen ’ s Kappa for some coded variables show how important it is not only to use consistent reporting standards but also to use coherent definitions of equivalent theoretical concepts and to describe study features as detailed as possible.”
MethodInfluentialA novel difficulty scoring system for laparoscopic liver resection
Ban, D., Tanabe, M., Ito, H., Otsuka, Y., Nitta, H., Abe, Y., Hasegawa, Y., Katagiri, T., Takagi, C., Itano, O., et al. (2014) · Journal of Hepato-Biliary-Pancreatic Sciences · doi:10.1002/jhbp.166 · Cited by 563
External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)
Shea, B. J., Bouter, L. M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V. K., & Grimshaw, J. M. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001350 · Cited by 569
“Kappa values of less than 0 rate as less than chance agreement; 0.01–0.20 slight agreement; 0.21– 0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 substantial agreement; and 0.81–0.99 almost perfect agreement [52], [57].”
BackgroundMethodInfluentialReliability from α to ω: A tutorial.
Revelle, W., & Condon, D. M. (2019) · Psychological Assessment · doi:10.1037/pas0000754 · Cited by 506
“If some disagreements are more important than others, we have weighted κ which with appropriate weights is equal to the intraclass correlation between the raters (Cohen, 1968; Fleiss & Cohen, 1973).”
BackgroundLaboratory diagnosis of iron-deficiency anemia: An overview
Guyatt, G. H., Oxman, A. D., Ali, M., Willan, A., McIlroy, W., & Patterson, C. (1992) · Journal of General Internal Medicine · doi:10.1007/bf02598003 · Cited by 564
Umer, A., Kelley, G. A., Cottrell, L. E., Giacobbi, P., Innes, K. E., & Lilly, C. L. (2017) · BMC Public Health · doi:10.1186/s12889-017-4691-z · Cited by 515
Systematic review and meta-analysis of augmented reality in educational settings
Garzón, J., Pavón, J., & Baldiris, S. (2019) · Virtual Reality · doi:10.1007/s10055-019-00379-9 · Cited by 562
“This value was found to be 0.94, which corresponds to almost perfect agreement as stated by Cohen (1968).”
ResultInfluentialGrassmann, F., Mengelkamp, J., Brandl, C., Harsch, S., Zimmermann, M. E., Linkohr, B., Peters, A., Heid, I. M., Palm, C., & Weber, B. H. F. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.02.037 · Cited by 549
Remission and active disease in rheumatoid arthritis: Defining criteria for disease activity states
Aletaha, D., Ward, M. M., Machold, K. P., Nell, V. P. K., Stamm, T., & Smolen, J. S. (2005) · Arthritis & Rheumatism · doi:10.1002/art.21235 · Cited by 542
Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., Lenderking, W. R., Cella, D., & Basch, E. (2009) · Value in Health · doi:10.1111/j.1524-4733.2008.00470.x · Cited by 537
Kappa coefficients in medical research
Chmura Kraemer, H., Periyakoil, V. S., & Noda, A. (2002) · Statistics in Medicine · doi:10.1002/sim.1180 · Cited by 527
“The original introductions of kappa [3; 40] de ned not the population parameter, , but the sample estimate k, where the probability of agreement is replaced by the observed proportion of agreement, and P is estimated by the proportion of the classi cations that selected category 1.”
BackgroundCommunication of Social Support in Computer-Mediated Groups for People With Disabilities
Braithwaite, D. O., Waldron, V. R., & Finn, J. (1999) · Health Communication · doi:10.1207/s15327027hc1102_2 · Cited by 528
How does tillage intensity affect soil organic carbon? A systematic review
Haddaway, N. R., Hedlund, K., Jackson, L. E., Kätterer, T., Lugato, E., Thomsen, I. K., Jørgensen, H. B., & Isberg, P.-E. (2017) · Environmental Evidence · doi:10.1186/s13750-017-0108-9 · Cited by 485
“A Kappa tests [42] for consistency checking were performed to assess the level of agreement amongst members of the review team (NRH, KH and HBJ), indicating high agreement at abstract (kappa = 0.”
MethodRumination and overgeneral memory in depression: Effects of self-focus and analytic thinking.
Watkins, E., & Teasdale, J. D. (2001) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.110.2.333 · Cited by 536
Radiological versus histological diagnosis in UIP and NSIP: survival implications
Flaherty, K. R. (2003) · Thorax · doi:10.1136/thorax.58.2.143 · Cited by 539
Body mass index and the risk of stroke in men
Kurth, T., Gaziano, J. M., Berger, K., Kase, C. S., Rexrode, K. M., Cook, N. R., Buring, J. E., & Manson, J. E. (2002) · Archives of Internal Medicine · doi:10.1001/archinte.162.22.2557 · Cited by 524
Neuman, S. B., & Cunningham, L. (2009) · American Educational Research Journal · doi:10.3102/0002831208328088 · Cited by 522
“Cohen’s kappa statistic (Cohen, 1960, 1968) was used to calculate reliability.”
MethodA disagreement about within-group agreement: disentangling issues of consistency versus consensus
Kozlowski, S. W., & Hattrup, K. (1992) · Journal of Applied Psychology · doi:10.1037/0021-9010.77.2.161 · Cited by 514
“Early writers often used the terms interrater reliabilily and interrater agreement interchangeably (e.g., Cohen, 1968; Finn, 1970).”
BackgroundMethodInfluentialThe Eras and Trends of Automatic Short Answer Grading
Burrows, S., Gurevych, I., & Stein, B. (2015) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0026-8 · Cited by 461
“Ratio Data (Discrete) Considering discrete ratio data in Table 8, the weighted variant Cohen’s κ is the metric applied (Cohen 1968).”
MethodUnderstanding Transference: The Core Conflictual Relationship Theme Method
Luborsky, L. B., & Crits-Christoph, P. F. (1990) · Medical Entomology and Zoology · Cited by 600
Measuring Agreement between Two Judges on the Presence or Absence of a Trait
Fleiss, J. L. (1975) · Biometrics · doi:10.2307/2529549 · Cited by 501
The Role of Home Literacy Practices in Preschool Children's Language and Emergent Literacy Skills
Roberts, J., Jergens, J., & Burchinal, M. (2005) · Journal of Speech, Language, and Hearing Research · doi:10.1044/1092-4388(2005/024 · Cited by 465
The Repeatability of Clinical Measurements of Dry Eye
Nichols, K. K., Mitchell, G. L., & Zadnik, K. (2004) · Cornea · doi:10.1097/00003226-200404000-00010 · Cited by 493
“A standard weighting system proposed by Fleiss and Cohen was used for ordered categorical data (ie, symptom assessment, Meibomian gland evaluation, and quantification of tear debris) to give “partial credit” for values adjacent to the expected values.(16,17) The 274 CORNEA”
MethodChicco, D., Warrens, M. J., & Jurman, G. (2021) · IEEE Access · doi:10.1109/access.2021.3084050 · Cited by 444
FOODY, G. M. (1996) · International Journal of Remote Sensing · doi:10.1080/01431169608948706 · Cited by 491
Handbook of multimethod measurement in psychology.
(2006) · American Psychological Association eBooks · doi:10.1037/11383-000 · Cited by 481
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
Blake, M. R., Raker, J. M., & Whelan, K. (2016) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.13746 · Cited by 495
Groenvold, M., Klee, M. C., Sprangers, M. A. G., & Aaronson, N. K. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00428-3 · Cited by 482
Meta-analysis of the impact of Augmented Reality on students’ learning gains
Garzón, J., & Acevedo, J. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.04.001 · Cited by 491
Masud, M., Sikder, N., Nahid, A.-A., Bairagi, A. K., & AlZain, M. A. (2021) · Sensors · doi:10.3390/s21030748 · Cited by 464
“Quadratic Weighted Kappa (QWK) is a chanceadjusted matric to judge the reliability of categorical measurements [50].”
BackgroundAnalysis and Comparison of the Psychometric Properties of Three Balance Measures for Stroke Patients
Mao, H.-F., Hsueh, I.-P., Tang, P.-F., Sheu, C.-F., & Hsieh, C.-L. (2002) · Stroke · doi:10.1161/01.str.0000012516.63191.c5 · Cited by 485
Dunning, J., Daly, J. P., Lomas, J.-P., Lecky, F., Batchelor, J., Mackway-Jones, K., & on behalf of the children's head injury algorithm for the prediction of important clinical events (CHALICE) study group (2006) · Archives of Disease in Childhood · doi:10.1136/adc.2005.083980 · Cited by 481
CASE SURVEY METHODOLOGY: QUANTITATIVE ANALYSIS OF PATTERNS ACROSS CASE STUDIES.
LARSSON, R. (1993) · Academy of Management Journal · doi:10.2307/256820 · Cited by 443
Feasibility, reliability, and validity of the EQ-5D-Y: results from a multinational study
Ravens-Sieberer, U., Wille, N., Badia, X., Bonsel, G., Burström, K., Cavrini, G., Devlin, N., Egmar, A.-C., Gusi, N., Herdman, M., et al. (2010) · Quality of Life Research · doi:10.1007/s11136-010-9649-x · Cited by 464
“The question about medication should indicate the severity of the condition reported [17].”
Monahan, P. O., Shacham, E., Reece, M., Kroenke, K., Ong’or, W. O., Omollo, O., Yebei, V. N., & Ojwang, C. (2009) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0846-z · Cited by 464
Development and validation of a clinical scale for the diagnosis of drug-induced hepatitis
Maria, V. A., & Victorino, R. M. (1997) · Hepatology · doi:10.1002/hep.510260319 · Cited by 351
The effects of argumentation scaffolds on argumentation and problem solving
Cho, K.-L., & Jonassen, D. H. (2002) · Educational Technology Research and Development · doi:10.1007/bf02505022 · Cited by 444
Recommendations for reporting the results of studies of instrument and scale development and testing
Streiner, D. L., & Kottner, J. (2014) · Journal of Advanced Nursing · doi:10.1111/jan.12402 · Cited by 463
“present/absent; mild/moderate/severe), Cohen’s kappa (Cohen 1960) or weighted kappa (Cohen 1968) are often used, but it should be remembered that the results are identical to some of the ICCs (Fleiss & Cohen 1973).”
ResultAdaptação transcultural da Escala de Independência em Atividades da Vida Diária (Escala de Katz)
Lino, V. T. S., Pereira, S. R. M., Camacho, L. A. B., Ribeiro Filho, S. T., & Buksman, S. (2008) · Cadernos de Saúde Pública · doi:10.1590/s0102-311x2008000100010 · Cited by 439
Statistical Techniques For Comparing Measurers And Methods Of Measurement: A Critical Review
Ludbrook, J. (2002) · Clinical and Experimental Pharmacology and Physiology · doi:10.1046/j.1440-1681.2002.03686.x · Cited by 455
“Cohen originally described an arbitrary weighting system, in which the investigator decides arbitrarily which disagreements should attract the worst ‘penalty’.(13) It is rarely used nowadays.”
BackgroundMethodInfluentialAssessing relationship quality in mandated community treatment: Blending care with control.
Skeem, J. L., Louden, J. E., Polaschek, D., & Camp, J. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.4.397 · Cited by 424
“On the basis of the last four training cases, the team's average reliability for DRI ratings was good (weighted kappa ϭ .65, SD ϭ .07; Cohen, 1968).”
Development, Use, and Psychometric Properties of the Trauma History Questionnaire
Hooper, L. M., Stockton, P., Krupnick, J. L., & Green, B. L. (2011) · Journal of Loss and Trauma · doi:10.1080/15325024.2011.572035 · Cited by 399
Dynamic adaptation process to implement an evidence-based child maltreatment intervention
Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., Silovsky, J. F., Hecht, D. B., & Chaffin, M. J. (2012) · Implementation Science · doi:10.1186/1748-5908-7-32 · Cited by 449
Reproducibility of a histopathologic grading system for adult soft tissue sarcoma
Coindre, J. M., Trojani, M., Contesso, G., David, M., Rouesse, J., Bui, N. B., Bodaert, A., De Mascarel, I., De Mascarel, A., & Goussot, J.-F. (1986) · Cancer · doi:10.1002/1097-0142(19860715)58:2<306::aid-cncr2820580216>3.0.co;2-7 · Cited by 449
Reproducibility of a histopathologic grading system for adult soft tissue sarcoma
Coindre, J., Trojani, M., Contesso, G., David, M., Rouëssé, J., Nguyen, B. B., Bodaert, A., de Mascarel, I., de Mascarel, A., & Goussot, J. (1986) · Cancer · doi:10.1002/1097-0142(19860715)58:2<222::aid-cncr2820580204>3.0.co;2-r · Cited by 448
Classification of Diabetic Retinopathy from Fluorescein Angiograms
(1991) · Ophthalmology · doi:10.1016/s0161-6420(13)38013-0 · Cited by 338
The reliability of self-assessed health status
Crossley, T. F., & Kennedy, S. (2002) · Journal of Health Economics · doi:10.1016/s0167-6296(02)00007-3 · Cited by 439
Prevalence and Risk Factors for Diabetes Mellitus in Nigeria: A Systematic Review and Meta-Analysis
Uloko, A. E., Musa, B. M., Ramalan, M. A., Gezawa, I. D., Puepet, F. H., Uloko, A. T., Borodo, M. M., & Sada, K. B. (2018) · Diabetes Therapy · doi:10.1007/s13300-018-0441-1 · Cited by 359
“Multiple coder agreement was assessed using Cohen’s kappa [9].”
MethodTomioka, K., Iwamoto, J., Saeki, K., & Okamoto, N. (2011) · Journal of Epidemiology · doi:10.2188/jea.je20110003 · Cited by 400
White, R. L., Vella, S., Biddle, S., Sutcliffe, J., Guagliano, J. M., Uddin, R., Burgin, A., Apostolopoulos, M., Nguyen, T., Young, C., et al. (2024) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-024-01676-6 · Cited by 298
Statistical Methods in Assessing Agreement: Models, Issues, and Tools
Lin, L., Hedayat, A. S., Sinha, B., & Yang, M. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502753479392 · Cited by 413
Masson, L., MCNeill, G., Tomany, J., Simpson, J., Peace, H., Wei, L., Grubb, D., & Bolton-Smith, C. (2003) · Public Health Nutrition · doi:10.1079/phn2002429 · Cited by 423
Modeling Species Distribution and Change Using Random Forest
Evans, J. S., Murphy, M. A., Holden, Z. A., & Cushman, S. A. (2011) · Predictive Species and Habitat Modeling in Landscape Ecology · doi:10.1007/978-1-4419-7390-0_8 · Cited by 364
“…does not account for the expected frequency of a class and does not make distinctions among various types and sources of disagreement the weighted Kappa was developed (Cohen 1968) The incorporation of a weight allows for near agreement and adjusts for expectation in the frequency of observations.”
Backgroundvan Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Interrater Reliability and Agreement
Tinsley, H. E. A., & Weiss, D. J. (2000) · Handbook of Applied Multivariate Statistics and Mathematical Modeling · doi:10.1016/b978-012691360-6/50005-7 · Cited by 394
Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review
Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2013.873278 · Cited by 409
“For calculating the interrelated agreement between two reviewers, the weighted Cohen’s kappa coefficient is determined (J. Cohen, 1968).”
MethodLarge sample variance of kappa in the case of different sets of raters.
Fleiss, J. L., Nee, J. C., & Landis, J. R. (1979) · Psychological Bulletin · doi:10.1037//0033-2909.86.5.974 · Cited by 398
A Self-Administered Questionnaire for Assessment of Symptoms and Function of the Shoulder*
L'insalata, J. C., Warren, R. F., Cohen, S. B., Altchek, D. W., & Peterson, M. G. E. (1997) · The Journal of Bone and Joint Surgery (American Volume) · doi:10.2106/00004623-199705000-00014 · Cited by 409
Super‐resolution musculoskeletal MRI using deep learning
Chaudhari, A. S., Fang, Z., Kogan, F., Wood, J., Stevens, K. J., Gibbons, E. K., Lee, J. H., Gold, G. E., & Hargreaves, B. A. (2018) · Magnetic Resonance in Medicine · doi:10.1002/mrm.27178 · Cited by 406
“41,42 All numerical analysis and data preprocessing were performed using MATLAB.”
MethodBrown, J. C., Huedo-Medina, T. B., Pescatello, L. S., Pescatello, S. M., Ferrer, R. A., & Johnson, B. T. (2011) · Cancer Epidemiology, Biomarkers & Prevention · doi:10.1158/1055-9965.epi-10-0988 · Cited by 404
Westphal, J. D., & Bednar, M. K. (2005) · Administrative Science Quarterly · doi:10.2189/asqu.2005.50.2.262 · Cited by 404
Seddon, J. M. (2003) · Archives of Ophthalmology · doi:10.1001/archopht.121.6.785 · Cited by 398
“High school education 70 (91) 105 (88) 53 (83) 71 (96) 70 (83) 71 (84) 74 (90) 69 (90) 69 (82) Initial AMD grade in worst eye 2 11 (14) 19 (16) 5 (8) 15 (20) 9 (11) 9 (11) 16 (20) 9 (12) 8 (10) 3 29 (38) 50 (42) 20 (31) 31 (42) 33 (39) 27 (32) 28 (34) 28 (36) 35 (42) 4 11 (14) 16 (13) 12 (19) 7 (9) 14 (17) 15 (18) 12…”
Vargas, H. A., Akin, O., Franiel, T., Mazaheri, Y., Zheng, J., Moskowitz, C., Udo, K., Eastham, J., & Hricak, H. (2011) · Radiology · doi:10.1148/radiol.11102066 · Cited by 402
Näsi, R., Honkavaara, E., Lyytikäinen-Saarenmaa, P., Blomqvist, M., Litkey, P., Hakala, T., Viljanen, N., Kantola, T., Tanhuanpää, T., & Holopainen, M. (2015) · Remote Sensing · doi:10.3390/rs71115467 · Cited by 381
“Based on this method, we calculated the confusion matrix, Cohen’s kappa value (κ) [47], the producer’s, user’s and overall accuracies.”
MethodEffects of virtual reality on learning outcomes in K-6 education: A meta-analysis
Villena-Taranilla, R., Tirado-Olivares, S., Cózar-Gutiérrez, R., & González-Calero, J. A. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100434 · Cited by 310
Zwicker, J. G., Harris, S. R., & Klassen, A. F. (2013) · Child: Care, Health and Development · doi:10.1111/j.1365-2214.2012.01379.x · Cited by 363
“We analysed inter-rater agreement for abstract and full-text screening using the kappa statistic (Cohen 1968).”
MethodVisser, H., & de Nijs, T. (2006) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2004.11.013 · Cited by 370
“Other measures were discussed by Cohen (1968); Turk (2002) and Foody (2004).”
BackgroundLambregts, D. M. J., Vandecaveye, V., Barbaro, B., Bakers, F. C. H., Lambrecht, M., Maas, M., Haustermans, K., Valentini, V., Beets, G. L., & Beets-Tan, R. G. H. (2011) · Annals of Surgical Oncology · doi:10.1245/s10434-011-1607-5 · Cited by 390
“81–1 excellent agreement) were calculated to evaluate interobserver variability.(22)”
MethodA global assessment of Indigenous community engagement in climate research
David-Chavez, D. M., & Gavin, M. C. (2018) · Environmental Research Letters · doi:10.1088/1748-9326/aaf300 · Cited by 332
“We included an inter-rater reliability test (kappa=0.907) using Cohen’s weighted kappa to assess for consistency in coded values (Cohen 1968).”
MethodInfluentialDel-Ben, C. M., Vilela, J. A. A., Crippa, J. A. D. S., Hallak, J. E. C., Labate, C. M., & Zuardi, A. W. (2001) · Revista Brasileira de Psiquiatria · doi:10.1590/s1516-44462001000300008 · Cited by 386
Balance and Sequence in Online Reviews: How Perceived Usefulness Affects Attitudes and Intentions
Purnawirawan, N., De Pelsmacker, P., & Dens, N. (2012) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2012.04.002 · Cited by 364
“Intercoder reliability, assessed by Cohen's κ score, was higher than .875 for all reviews, while values of .80 or higher indicate sufficient reliability (+1.0 indicates perfect agreement, 0.0 indicates no agreement other than expected by chance) (Bakeman 2000; Cohen 1968).”
MethodInfluentialSeddon, J. M. (2003) · Archives of Ophthalmology · doi:10.1001/archopht.121.12.1728 · Cited by 384
Judging the quality of care at the end of life: can proxies provide reliable information?
McPherson, C. J., & Addington-Hall, J. M. (2003) · Social Science & Medicine · doi:10.1016/s0277-9536(02)00011-4 · Cited by 378
Akin, O., Sala, E., Moskowitz, C. S., Kuroiwa, K., Ishill, N. M., Pucar, D., Scardino, P. T., & Hricak, H. (2006) · Radiology · doi:10.1148/radiol.2392050949 · Cited by 379
“,5 denote the rating categories for the readers (21,22).”
BackgroundMariel, P., Hoyos, D., Meyerhoff, J., Czajkowski, M., Dekker, T., Glenk, K., Jacobsen, J. B., Liebe, U., Olsen, S. B., Sagebiel, J., et al. (2021) · SpringerBriefs in Economics · doi:10.1007/978-3-030-62669-3 · Cited by 233
Parsons, C. L., Dell, J., Stanford, E. J., Bullen, M., Kahn, B. S., Waxell, T., & Koziol, J. A. (2002) · Urology · doi:10.1016/s0090-4295(02)01829-0 · Cited by 383
Capturing quality in early childhood through environmental rating scales
Sylva, K., Siraj-Blatchford, I., Taggart, B., Sammons, P., Melhuish, E., Elliot, K., & Totsika, V. (2006) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2006.01.003 · Cited by 342
“It was calculated in two ways: (a) as the percentage of exact agreement between the two observers and (b) as a kappa coefficient (Cohen, 1968).”
MethodBwangoy, J.-R. B., Hansen, M. C., Roy, D. P., Grandi, G. D., & Justice, C. O. (2010) · Remote Sensing of Environment · doi:10.1016/j.rse.2009.08.004 · Cited by 364
“The Kappa coefficient measures the agreement between variables by adjusting the percentage of agreement to the amount of agreement due to chance alone (Cohen, 1968), although Kappa values may be biased in areas with uneven proportions of the different classes (Foody, 2004; Stehman, 2006).”
BackgroundSiragusa, S., Cosmi, B., Piovella, F., Hirsh, J., & Ginsberg, J. S. (1996) · The American Journal of Medicine · doi:10.1016/s0002-9343(97)89484-3 · Cited by 373
Atherosclerosis influences the vasomotor response of epicardial coronary arteries to exercise.
Gordon, J. B., Ganz, P., Nabel, E. G., Fish, R. D., Zebede, J., Mudge, G. H., Alexander, R. W., & Selwyn, A. P. (1989) · Journal of Clinical Investigation · doi:10.1172/jci114103 · Cited by 367
Vasbinder, G. B. C., Nelemans, P. J., Kessels, A. G. H., Kroon, A. A., Maki, J. H., Leiner, T., Beek, F. J. A., Korst, M. B. J. M., Flobbe, K., de Haan, M. W., et al. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-141-9-200411020-00007 · Cited by 365
“Intra-arterial DSA results, n (%) Renal artery stenosis 72 ( 20 )”
BackgroundPanel, S. G. P. C., Hoekstra, F., Mrklas, K. J., Khan, M., McKay, R. C., Vis-Dunbar, M., Sibley, K. M., Nguyen, T., Graham, I. D., & Gainforth, H. L. (2020) · Health Research Policy and Systems · doi:10.1186/s12961-020-0544-9 · Cited by 343
Tan, Y., & Cahan, P. (2019) · Cell Systems · doi:10.1016/j.cels.2019.06.004 · Cited by 368
Validating Expert System Performance
O'Keefe, R. M., Balci, O., & Smith, E. P. (1987) · IEEE Expert · doi:10.1109/mex.1987.5006538 · Cited by 358
Does Self-Control Training Improve Self-Control? A Meta-Analysis
Friese, M., Frankenbach, J., Job, V., & Loschelder, D. D. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617697076 · Cited by 334
Vitali, C., Moutsopoulos, H. M., & Bombardieri, S. (1994) · Annals of the Rheumatic Diseases · doi:10.1136/ard.53.10.637 · Cited by 364
The Canadian Neurological Scale: a preliminary study in acute stroke.
Côté, R., Hachinski, V. C., Shurvell, B. L., Norris, J. W., & Wolfson, C. (1986) · Stroke · doi:10.1161/01.str.17.4.731 · Cited by 341
de Dreu, M. J., van der Wilk, A. S. D., Poppe, E., Kwakkel, G., & van Wegen, E. E. H. (2012) · Parkinsonism & Related Disorders · doi:10.1016/s1353-8020(11)70036-0 · Cited by 322
“Cohen’s kappa was calculated for inter-rater reliability [15].”
MethodDifficulties and Coping Strategies of Sudanese Refugees: A Qualitative Approach
Khawaja, N. G., White, K. M., Schweitzer, R., & Greenslade, J. (2008) · Transcultural Psychiatry · doi:10.1177/1363461508094678 · Cited by 307
“Cohen’s kappa (Cohen, 1968) was used to calculate inter-rater agreement for the analysed themes.”
MethodEffect of Local Cultural Context on the Success of Community‐Based Conservation Interventions
WAYLEN, K. A., FISCHER, A., MCGOWAN, P. J. K., THIRGOOD, S. J., & MILNER‐GULLAND, E. J. (2010) · Conservation Biology · doi:10.1111/j.1523-1739.2010.01446.x · Cited by 318
“(Cohen 1968; Siegel & Castellan 1988).”
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.