Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 701–800 of 1,098
Measures of interobserver agreement: Calculation formulas and distribution effects
House, A. E. (1982) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/bf01321270 · Cited by 128
Schizo-affective psychosis: definitions and incidence
Brockington, I. F., & Leff, J. P. (1979) · Psychological Medicine · doi:10.1017/s0033291700021590 · Cited by 129
Lahaye, M. J., Beets, G. L., Engelen, S. M. E., Kessels, A. G. H., de Bruı̈ne, A. P., Kwee, H., van Engelshoven, J. M. A., van de Velde, C. J. �., & Beets‐Tan, R. G. H. (2009) · Radiology · doi:10.1148/radiol.2521081364 · Cited by 128
“To determine the interobserver agreement, weighted values (21,22) were calculated for the ordinal variables and intraclass correlations for interval-scaled variables (23), all with their 95% confidence intervals.”
MethodAgreement Between Retrospective Accounts of Substance Use and Earlier Reported Substance Use
Collins, L. M., Graham, J. W., Hansen, W. B., & Johnson, C. A. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900308 · Cited by 125
Manual and automatic locomotion scoring systems in dairy cows: A review
Schlageter-Tello, A., Bokkers, E. A. M., Koerkamp, P. W. G. G., van Hertem, T., Viazzi, S., Romanini, C. E. B., Halachmi, I., Bahr, C., Berckmans, D., & Lokhorst, K. (2014) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2014.06.006 · Cited by 152
Bhatt, A., Ganatra, A. P., & Kotecha, K. V. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.348 · Cited by 141
“Recall ¼ We used Cohen ’ s Kappa Score ( Cohen, 1960 , 1968 ), which is a statistical measure for measuring “ Intra-rater Reliability ” and is a more robust measure than simple percent agreement calculation.”
MethodWells, A. U., Hansell, D. M., HASLAM, P. L., Rubens, M. B., Cailes, J., Black, C. M., & du Bois, R. M. (1998) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.157.5.9609096 · Cited by 127
Intercoder reliability indices: disuse, misuse, and abuse
Feng, G. C. (2013) · Quality & Quantity · doi:10.1007/s11135-013-9956-8 · Cited by 142
“6 Cohen (1968) later proposed weighted κ for ordinal ratings.”
BackgroundLanguage and Play at One Year: A Comparison of Toddlers and Mothers in the United States and Japan
Tamis‐LeMonda, C. S., Bornstein, M. H., Cyphers, L. H., Toda, S., & Ogino, M. (1992) · International Journal of Behavioral Development · doi:10.1177/016502549201500102 · Cited by 127
van der Meulen, I., van de Sandt‐Koenderman, W. M. E., Duivenvoorden, H. J., & Ribbers, G. M. (2009) · International Journal of Language & Communication Disorders · doi:10.3109/13682820903111952 · Cited by 141
ATRX immunostaining predicts IDH and H3F3A status in gliomas
Ebrahimi, A., Skardelly, M., Bonzheim, I., Ott, I., Mühleisen, H., Eckert, F., Tabatabai, G., & Schittenhelm, J. (2016) · Acta Neuropathologica Communications · doi:10.1186/s40478-016-0331-6 · Cited by 134
“To assess the potential interobserver variability, we performed a Pearsons Chi-square test on the data collected by the two neuropathologists and calculated Cohen's kappa coefficient for agreement [18].”
MethodKelley, G. A., Kelley, K. S., Roberts, S., & Haskell, W. (2012) · Clinical Nutrition · doi:10.1016/j.clnu.2011.11.011 · Cited by 125
Nailfold capillary patterns in healthy subjects: A real issue in capillaroscopy
Ingegnoli, F., Gualtierotti, R., Lubatti, C., Bertolazzi, C., Gutiérrez, M., Boracchi, P., Fornili, M., & Angelis, R. D. (2013) · Microvascular Research · doi:10.1016/j.mvr.2013.07.001 · Cited by 125
“The inter-observer agreement was estimated by the weighted Cohen's kappa (WCK) for the total number of capillaries and the number of capillaries of each morphological type (Cohen, 1968) and by the Intraclass Correlation Index (ICC) for the various capillary length measurements (Bartko, 1966).”
MethodRequirements engineering tools: Capabilities, survey and assessment
de Gea, J. M. C., Nicolás, J., Fernández‐Alemán, J. L., Toval, A., Ebert, C., & Vizcaíno, A. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.04.005 · Cited by 125
“Since we start from ordered codes (Very high, High, Medium, Low and Very low) and Weighted Kappa [35] allows different levels of agreement to contribute to the value of Kappa when the categories are ordered (i.e. when the variable is ordinal), this method was eventually applied.”
BackgroundLiu, Z., Que, S., Zhou, L., & Zheng, S. (2015) · Scientific Reports · doi:10.1038/srep14325 · Cited by 132
Weighted kappa loss function for multi-class classification of ordinal data in deep learning
de la Torre, J., Puig, D., & Valls, A. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.05.018 · Cited by 151
Koos Classification of Vestibular Schwannomas: A Reliability Study
Erickson, N. J., Schmalz, P. G. R., Agee, B. S., Fort, M., Walters, B. C., McGrew, B. M., & Fisher, W. S. (2018) · Neurosurgery · doi:10.1093/neuros/nyy409 · Cited by 136
An Independent Evaluation of Four Quantitative Emergency Department Crowding Scales
Jones, S. S., Allen, T. L., Flottemesch, T. J., & Welch, S. J. (2006) · Academic Emergency Medicine · doi:10.1197/j.aem.2006.05.021 · Cited by 125
Logic and reliability of evaluations of competence to stand trial.
Skeem, J. L., Golding, S. L., Cohn, N. B., & Berge, G. (1998) · Law and Human Behavior · doi:10.1023/a:1025787429972 · Cited by 125
“For the rating variables, two forms of interrater reliability were computed: weighted kappa (Cohen, 1968)16 and percent agreement.”
MethodEye drop instillation technique in patients with glaucoma
Tatham, A. J., Sarodia, U., Gatrad, F., & Awan, A. Z. (2013) · Eye · doi:10.1038/eye.2013.187 · Cited by 125
Sausville, E. A., Worsham, G. F., Matthews, M. J., Makuch, R., Fischmann, A. B., Schechter, G., Gazdar, A. F., & Bunn, P. A. (1985) · Human Pathology · doi:10.1016/s0046-8177(85)80177-5 · Cited by 125
Asbestosis and Idiopathic Pulmonary Fibrosis: Comparison of Thin-Section CT Features
Copley, S. J., Wells, A. U., Sivakumaran, P., Rubens, M. B., Gary Lee, Y. C., Desai, S. R., MacDonald, S. L. S., Thompson, R. I., Colby, T. V., Nicholson, A. G., et al. (2003) · Radiology · doi:10.1148/radiol.2293020668 · Cited by 129
Berg, A. T., Levy, S. R., Testa, F. M., & Shinnar, S. (1999) · Epilepsia · doi:10.1111/j.1528-1157.1999.tb00738.x · Cited by 126
Suspected scaphoid fractures. The value of radiographs
Dias, J., Thompson, J., Barton, N., & Gregg, P. (1990) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.72b1.2298805 · Cited by 125
Optimum duration of regimens for Helicobacter pylori eradication
Yuan, Y., Ford, A. C., Khan, K. J., Forman, D., Oderda, G., Calvet, X., Leontiadis, G. I., Fallone, C. A., Gisbert, J. P., Fischbach, L. A., et al. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008337 · Cited by 125
Armijo-Olivo, S., Ospina, M., da Costa, B. R., Egger, M., Saltaji, H., Fuentes, J., Ha, C., & Cummings, G. G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0096920 · Cited by 124
“Statistical analysis Inter-rater reliability of RoB assessments between Cochrane and blinded external reviewers [18–20] and within the panel of external reviewers was assessed using weighted kappa (k) for categorical data.”
Methodde Oliveira, F. A., Luna, S. P. L., do Amaral, J. B., Rodrigues, K. A., Sant’Anna, A. C., Daolio, M., & Brondani, J. T. (2014) · BMC Veterinary Research · doi:10.1186/s12917-014-0200-0 · Cited by 124
“The weighted kappa coefficient was then calculated with a 95% confidence interval (CI) [40] for each scale item, considering all of the assessment times as a group (MA =M1, M2, M3 and M4).”
MethodBhandari, R., Kelley, G. A., Hartley, T. A., & Rockett, I. R. H. (2014) · International Journal of Breast Cancer · doi:10.1155/2014/189384 · Cited by 118
“Using Cohen’s kappa (k) statistic [66], the overall interrater agreement rate prior to correcting discrepant items was 0.”
MethodLearning to Think Critically: A Visual Art Experiment
Bowen, D. H., Greene, J. P., & Kisida, B. (2013) · Educational Researcher · doi:10.3102/0013189x13512675 · Cited by 164
“Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.”
BackgroundMethodInfluentialFlodén, J. (2025) · British Educational Research Journal · doi:10.1002/berj.4069 · Cited by 82
Warren, A. R., Woodall, C. E., Hunt, J. S., & Perry, N. W. (1996) · Child Maltreatment · doi:10.1177/1077559596001003006 · Cited by 124
Strachman, A., & Gable, S. L. (2006) · Personality and Social Psychology Bulletin · doi:10.1177/0146167206291007 · Cited by 115
“Weighted kappa ( Cohen, 1968 ) was computed to determine interrater reliability for the included variable.”
MethodSong, J., Semanik, P., Sharma, L., Chang, R. W., Hochberg, M. C., Mysiw, W. J., Bathon, J. M., Eaton, C. B., Jackson, R., Kwoh, C. K., et al. (2010) · Arthritis Care & Research · doi:10.1002/acr.20305 · Cited by 118
“the application of different threshold values (26).”
Accuracy and Reliability of Remote Retinopathy of Prematurity Diagnosis
Chiang, M. F., Keenan, J. D., Starren, J. B., Du, Y. E., Schiff, W. M., Barile, G. R., Li, J., Johnson, R. A., Hess, D. J., & Flynn, J. T. (2006) · Archives of Ophthalmology · doi:10.1001/archopht.124.3.322 · Cited by 128
Yatani (2012) · doi:10.1145/2207676.2207734 · Cited by 123
“To measure the inter-rater reliability of this ordinal scale, we calculated the Cohen’s κ with the squared weighting [4].”
MethodAtopic dermatitis symptoms decreased in children following massage therapy.
Schachner, L., Field, T., Hernandez-Reif, M., Duarte, A., & Krasnegor, J. (1998) · Pediatric Dermatology · doi:10.1046/j.1525-1470.1998.1998015390.x · Cited by 125
Regional differences in the severity of Lewy body pathology across the olfactory cortex
Silveira‐Moriyama, L., Holton, J. L., Kingsbury, A. E., Ayling, H., Petrie, A., Sterlacci, W., Poewe, W. H., Maier, H., Lees, A. J., & Révész, T. (2009) · Neuroscience Letters · doi:10.1016/j.neulet.2009.02.006 · Cited by 130
Ruggeri, M., Lasalvia, A., Dall'Agnola, R., Van Wijngaarden, B., Knudsen, H. C., Leese, M., Gaite, L., Tansella, M., & EPSILON Study Group (2000) · British Journal of Psychiatry · doi:10.1192/bjp.177.39.s41 · Cited by 122
Canaway, A., & Frew, E. J. (2012) · Quality of Life Research · doi:10.1007/s11136-012-0119-5 · Cited by 135
A Dictionary-based Approach to Racism Detection in Dutch Social MediaPreprint
Tulkens, S., Hilte, L., Lodewyckx, E., Verhoeven, B., & Daelemans, W. M. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.08738 · Cited by 123
“We calculated inter-annotator agreement using the Kappa score () (Cohen, 1968).”
MethodAssessing the inter-rater agreement for ordinal data through weighted indexes
Marasini, D., Quatto, P., & Ripamonti, E. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280214529560 · Cited by 115
Evaluation of Spectral Indices for Assessing Fire Severity in Australian Temperate Forests
Tran, B. N., Tanase, M. A., Bennett, L. T., & Aponte, C. (2018) · Remote Sensing · doi:10.3390/rs10111680 · Cited by 123
“In addition, Cohen’s kappa statistic, k, was used to assess the agreement between the reference and predicted classes [68].”
MethodLiterature and Practice: A Critical Review of MOOCs
Chiappe-Laverde, A., Hine, N., & Martínez-Silva, J. A. (2014) · Comunicar · doi:10.3916/c44-2015-01 · Cited by 150
“…of the items, the reading was conducted by two different observers, who separately identified key topics or concepts presented in each text which were compared using the Cohen´s Kappa coefficient (Cohen, 1968) from which observational consistency was established (Gordillo & Rodríguez, 2009).”
MethodHeart Rate Variability and Sustained Attention in ADHD Children
Börger, N. A., van der Meere, J., Ronner, A. E., Alberts, E., Geuze, R. H., & Bogte, H. (1999) · Research on Child and Adolescent Psychopathology · doi:10.1023/a:1022610306984 · Cited by 138
Assessment of agreement among several raters formulating multiple diagnoses
Mezzich, J. E., KRAEMER, H. C., Worthington, D. R. L., & Coffman, G. A. (1981) · Journal of Psychiatric Research · doi:10.1016/0022-3956(81)90011-x · Cited by 122
Prchal, A., & Landolt, M. A. (2012) · Cancer Nursing · doi:10.1097/ncc.0b013e31821e0c59 · Cited by 101
Aickin, M. (1990) · Biometrics · doi:10.2307/2531434 · Cited by 122
Miller, C. C., Glick, W. H., Wang, Y.-D., & Huber, G. P. (1991) · Academy of Management Journal · doi:10.2307/256447 · Cited by 119
(2013) · Cited by 121
Sharing the Emotional Load: Recipient Affiliation Calms Down the Storyteller
Peräkylä, A., Henttonen, P., Voutilainen, L., Kahri, M., Stevanovic, M., Sams, M. E., & Ravaja, N. (2015) · Social Psychology Quarterly · doi:10.1177/0190272515611054 · Cited by 144
Does McNemar’s test compare the sensitivities and specificities of two diagnostic tests?
Kim, S., & Lee, W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214541852 · Cited by 120
Antibiotic Prophylaxis for Urinary Tract Infections in Antenatal Hydronephrosis
Braga, L. H., Mijović, H., Farrokhyar, F., Pemberton, J., DeMaria, J., & Lorenzo, A. J. (2012) · PEDIATRICS · doi:10.1542/peds.2012-1870 · Cited by 120
An analysis of electronic health record–related patient safety incidents
Palojoki, S., Mäkelä, M., Lehtonen, L., & Saranto, K. (2017) · Health Informatics Journal · doi:10.1177/1460458216631072 · Cited by 108
Inoue, M., Jung, J. J., Balaratnasingam, C., Dansingani, K. K., Dhrami‐Gavazi, E., Suzuki, M., de Carlo, T. E., Shahlaee, A., Klufas, M. A., Maftouhi, A. E., et al. (2016) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.15-18900 · Cited by 120
“40 represents ‘‘poor’’ agreement.(23) The interobserver reproducibility of en face OCTA and FA was compared using Fisher’s exact test.”
MethodA Richly Annotated Corpus for Different Tasks in Automated Fact-Checking
Hanselowski, A., Stab, C., Schulz, C., Li, Z., & Gurevych, I. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1046 · Cited by 84
“7 (Cohen, 1968), indicating a substantial agreement between the crowd workers (Artstein and Poesio, 2008).”
BackgroundBenchmarking Kappa: Interrater Agreement in Software Process Assessments
Emam, K. E. (1999) · Empirical Software Engineering · doi:10.1023/a:1009820201126 · Cited by 120
Levin, M. F., Desrosiers, J., Beauchemin, D., Bergeron, N., & Rochette, A. (2004) · Physical Therapy · doi:10.1093/ptj/84.1.8 · Cited by 142
Langlois, Y., Roederer, G. O., Chan, A., Phillips, D. J., Beach, K. W., Martin, D., Chikos, P. M., & Strandness, D. E. (1983) · Ultrasound in Medicine & Biology · doi:10.1016/0301-5629(83)90109-6 · Cited by 120
A longitudinal content analysis of gender role portrayal in Belgian television advertising
Verhellen, Y., Dens, N., & Pelsmacker, P. D. (2014) · Journal of Marketing Communications · doi:10.1080/13527266.2013.871321 · Cited by 119
“After the data input by the two independent coders, we calculated the inter-coder reliability for all variables in the coding scheme using Cohen’s k coefficient (Cohen 1968).”
Methodde Felício, C. M., Medeiros, A. P. M., & Melchior, M. D. O. (2012) · Journal of Oral Rehabilitation · doi:10.1111/j.1365-2842.2012.02336.x · Cited by 133
Makris, T., Dorstyn, D., & Crettenden, A. (2019) · Disability and Rehabilitation · doi:10.1080/09638288.2019.1623852 · Cited by 127
Grading of Anterior Cruciate Ligament Injury
Hong, S. H., Choi, J., Lee, G. K., Choi, J.-A., Chung, H. W., & Kang, H. S. (2003) · Journal of Computer Assisted Tomography · doi:10.1097/00004728-200309000-00022 · Cited by 119
“Weighted statistics were used to analyze the diagnostic accuracies of MR imaging protocols A and B.(12) Interobserver agreement was also calculated using weighted statistics.”
MethodGlasgow Outcome Scale: An inter-rater reliability study
Anderson, S. I., Housley, A. M., Jones, P. A., Slattery, J., & Miller, J. D. (1993) · Brain Injury · doi:10.3109/02699059309034957 · Cited by 120
Østbye, T., Tyas, S. L., McDowell, I., & Koval, J. J. (1997) · Age and Ageing · doi:10.1093/ageing/26.2.99 · Cited by 119
“The weights are those proposed by Cohen [22] and Cicchetti and Allison [23].”
MethodRisk factors associated with mortality at a milk-fed veal calf facility: A prospective cohort study
Renaud, D. L., Duffield, T. F., LeBlanc, S. J., Ferguson, S., Haley, D. B., & Kelton, D. F. (2018) · Journal of Dairy Science · doi:10.3168/jds.2017-13581 · Cited by 118
“Using scores gathered from all calves arriving at the facility on June 17, 2016, inter- and intraobserver agreement were calculated for observers 1 and 2 using percentage agreement (McHugh, 2012) and weighted kappa (κ; Cohen, 1968).”
MethodKelley, G. A., & Kelley, K. S. (2013) · Journal of Obesity · doi:10.1155/2013/783103 · Cited by 103
“Using Cohen’s kappa statistic (κ) [21], the overall agreement rate prior to correcting discrepancies was 0.”
MethodBrendgen, M., Markiewicz, D., Doyle, A.-B., & Bukowski, W. M. (2001) · Merrill-Palmer Quarterly · doi:10.1353/mpq.2001.0013 · Cited by 107
“Interrater agreement for the various behavioral categories as reflected by weighted kappas (Cohen, 1968), based on the 25% of sessions that were rated by both coders, ranged from kappa .64 for criticism to kappa .85 for self-disclosure.”
Litman, D. J., & Forbes-Riley, K. (2005) · Speech Communication · doi:10.1016/j.specom.2005.09.008 · Cited by 136
“17 Since our emotion categories are ordinal/interval rather than nominal, we can also quantify agreement using a weighted version of Kappa (Cohen, 1968), which accounts for the relative distances between successive categories.”
MethodApples and Oranges: An Empirical Comparison of Commonly Used Indices of Interrater Agreement.
Jones, A. P., Johnson, L. A., Butler, M. C., & Main, D. S. (1983) · Academy of Management Journal · doi:10.2307/256262 · Cited by 122
Enhancing the self-esteem of youth swimmers through coach training: Gender and age effects
Coatsworth, J. D., & Conroy, D. E. (2005) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2005.08.005 · Cited by 160
Medical literature searches: a comparison of PubMed and Google Scholar
Nourbakhsh, E., Nugent, R., Wang, H., Cevik, C., & Nugent, K. (2012) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2012.00992.x · Cited by 113
Automated Scoring of Constructed‐Response Science Items: Prospects and Obstacles
Liu, O. L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., & Linn, M. C. (2014) · Educational Measurement: Issues and Practice · doi:10.1111/emip.12028 · Cited by 112
“We followed the Cohen (1968) rules for classification: none (0–.09), small (.10–.30), moderate (.31–.50), and large (.51–1.00).”
BackgroundMethodInfluentialNLP-driven citation analysis for scientometrics
JHA, R., JBARA, A.-A., QAZVINIAN, V., & RADEV, D. R. (2017) · Natural Language Engineering · doi:10.1017/s1351324915000443 · Cited by 105
“We used the Kappa coefficient (Cohen 1968) to measure the agreement.”
MethodFamily Boundary Ambiguity Among Transgender Youth
Catalpa, J. M., & McGuire, J. K. (2018) · Family Relations · doi:10.1111/fare.12304 · Cited by 128
Diabetic macular edema assessed with optical coherence tomography and stereo fundus photography.
Strøm, C., Sander, B., Larsen, N. B., Larsen, M., & Lund‐Andersen, H. (2002) · PubMed · Cited by 117
Bourgeois, M. S., Dijkstra, K., Burgio, L. D., & Allen, R. S. (2001) · Augmentative and Alternative Communication · doi:10.1080/aac.17.3.196.210 · Cited by 138
A Unified Approach for Assessing Agreement for Continuous and Categorical Data
Lin, L., Hedayat, A. S., & Wu, W. (2007) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543400701376498 · Cited by 114
Knowledge and Productivity in Technical Support Work
Das, A. (2003) · Management Science · doi:10.1287/mnsc.49.4.416.14419 · Cited by 128
“9 Cohen’s is defined as = PO −PC / 1−PC , where PO denotes the observed proportion of agreement and PC the proportion of agreement expected by chance (Cohen 1968).”
BackgroundDevelopment and Reliability of an Upper Extremity Function Test for the Elderly: The TEMPA
Desrosiers, J., Hébert, R., Dutil, É., & Bravo, G. (1993) · Canadian Journal of Occupational Therapy · doi:10.1177/000841749306000104 · Cited by 116
Hattori, M., & Oaksford, M. (2007) · Cognitive Science · doi:10.1080/03640210701530755 · Cited by 125
“(8) This index has been used as a measure of " response agreement " in psychology (Cohen, 1960, 1968; Fleiss, 1971; Light, 1971), " observer agreement " in clinical medicine (Landis & Koch, 1977), and the " accuracy " of weather forecasting (called the Heidke skill score) especially in America…”
MethodChanges in Chest Roentgenogram of Sarcoidosis Patients During a Clinical Trial of Infliximab Therapy
Baughman, R. P., Shipley, R. T., Desai, S. R., Drent, M., Judson, M. A., Costabel, U., du Bois, R. M., Kavuru, M. S., Schlenker‐Herceg, R., Flavin, S. K., et al. (2009) · CHEST Journal · doi:10.1378/chest.08-1876 · Cited by 120
Intonation Patterns in Child-Directed Speech: Mother-Father Differences
Warren-Leubecker, A., & III, J. N. B. (1984) · Child Development · doi:10.2307/1130007 · Cited by 113
“83 was established, using Cohen's weighted K (Cohen, 1968), which yields percentage agreement corrected for chance.”
MethodHefti, A. F. (1997) · Critical Reviews in Oral Biology & Medicine · doi:10.1177/10454411970080030601 · Cited by 146
Information From Relatives: Diagnosis of Affective Disorders
Gershon, E. S. (1984) · Archives of General Psychiatry · doi:10.1001/archpsyc.1984.01790130069010 · Cited by 115
Field, T., Healy, B., & LeBlanc, W. G. (1989) · Infant Behavior and Development · doi:10.1016/0163-6383(89)90044-1 · Cited by 115
Bryant, M. J., Ward, D. S., Hales, D., Vaughn, A., Tabak, R. G., & Stevens, J. (2008) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-5-23 · Cited by 110
Watts, F. M., & Finkenstaedt-Quinn, S. A. (2021) · Chemistry Education Research and Practice · doi:10.1039/d1rp00007a · Cited by 120
“…limitations: weighted kappa allows for ratio or interval coding, Fleiss’ kappa and Light’s kappa each allow for more than two researchers, and fuzzy kappa allows for multiple codes to be applied to each unit of analysis (Cohen, 1968; Fleiss, 1971; Light, 1971; Kirilenko and Stepchenkova, 2016).”
Generative AI-enhanced human-AI collaborative conceptual design: A systematic literature review
Fang, C., Zhu, Y., Fang, L., Long, Y., Lin, H., Cong, Y., & Wang, S. J. (2025) · Design Studies · doi:10.1016/j.destud.2025.101300 · Cited by 135
Cooper, J. E., Copeland, J. R. M., Brown, G. W., Harris, T., & Gourlay, A. J. (1977) · Psychological Medicine · doi:10.1017/s0033291700004499 · Cited by 116
Morfeld, K. A., Meehan, C. L., Hogan, J. N., & Brown, J. L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0155146 · Cited by 114
Williams, T. A., & Shepherd, D. A. (2017) · Journal of Management Studies · doi:10.1111/joms.12291 · Cited by 122
Paruzel, A., Klug, H. J. P., & Maier, G. W. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.607108 · Cited by 121
“Cohen’s kappa was computed for categorical data (Cohen, 1968): study design, subject group, and CSR focus and outcomemeasure and ranged from κ = 0.97 to κ = 1.”
MethodInfluentialOrtega, F. B., Sánchez‐López, M., Solera-Martínez, M., Fernández‐Sánchez, A., Sjöstróm, M., & Martínez‐Vizcaíno, V. (2012) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/j.1600-0838.2012.01454.x · Cited by 116
A repeated measures concordance correlation coefficient
King, T. S., Chinchilli, V. M., & Carrasco, J. L. (2007) · Statistics in Medicine · doi:10.1002/sim.2778 · Cited by 109
Causes of Pneumonia Epizootics among Bighorn Sheep, Western United States, 2008–2010
Besser, T. E., Highland, M. A., Baker, K. N. K., Cassirer, E. F., Anderson, N. J., Ramsey, J., Mansfield, K. G., Bruning, D. L., Wolff, P. L., Smith, J. B., et al. (2012) · Emerging infectious diseases · doi:10.3201/eid1803.111554 · Cited by 114
Barr, R. D., Simpson, T., Whitton, A. C., Rush, B., Furlong, W. J., & Feeny, D. H. (1999) · European Journal of Cancer · doi:10.1016/s0959-8049(98)00366-9 · Cited by 114
Khan, K. S., Daya, S., Collins, J. A., & Walter, S. D. (1996) · Fertility and Sterility · doi:10.1016/s0015-0282(16)58265-x · Cited by 126
Chmielewski, T. L., Hodges, M. J., Horodyski, M., Bishop, M. D., Conrad, B. P., & Tillman, S. M. (2007) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2007.2457 · Cited by 113
Multisite assessment of NIA‐AA guidelines for the neuropathologic evaluation of Alzheimer's disease
Montine, T. J., Monsell, S. E., Beach, T. G., Bigio, E. H., Bu, Y., Cairns, N. J., Frosch, M., Henriksen, J., Kofler, J., Kukull, W. A., et al. (2016) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2015.07.492 · Cited by 113
Assumptions behind Intercoder Reliability Indices
Zhao, X., Liu, J. S., & Deng, K. (2013) · Annals of the International Communication Association · doi:10.1080/23808985.2013.11679142 · Cited by 113
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.