Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 301–400 of 1,098
Johnson, T. D., & Joshi, A. (2016) · Journal of Applied Psychology · doi:10.1037/apl0000058 · Cited by 228
Orstad, S. L., McDonough, M. H., Stapleton, S., Altincekic, C., & Troped, P. J. (2017) · Environment and Behavior · doi:10.1177/0013916516670982 · Cited by 234
“We calculated frequencies for significant coefficients and for kappa scores (Cohen, 1968) based on the following agreement classifications (Landis & Koch, 1977): “poor” (κ .00), “slight” (κ = .00-.20), “fair” (κ = .21-.40), “moderate” (κ = .41-.60), “substantial” (κ = .61-.80), and “almost…”
MethodIntraepidermal nerve fiber density and its application in sarcoidosis
Bakkers, M., Merkies, I. S. J., Lauria, G., Devigili, G., Penza, P., Lombardi, R., Hermans, M. C. E., van Nes, S. I., De Baets, M., & Faber, C. G. (2009) · Neurology · doi:10.1212/wnl.0b013e3181bacf05 · Cited by 250
“Reasons for exclusion were immunosuppressive treatment (65), comorbidity that could cause neuropathy (30), CNS involvement (11) and large nerve fiber involvement found at neurologic or nerve conduction studies (21), and inability or unwillingness to attend examination (9).”
BackgroundNeural-machine-translation-based commit message generation: how far are we?
Liu, Z., Xia, X., Hassan, A. E., Lo, D., Xing, Z., & Wang, X. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238190 · Cited by 247
“Jiang We find a high level of agreement between the two raters with a Cohen’s Kappa coefficient [15] of 0.67, which shows a substantial agreement among the different raters.”
BackgroundInfluentialEkselius, L., Lindström, E., von Knorring, L., Bodlund, O., & Kullgren, G. (1994) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1994.tb01566.x · Cited by 246
Memories of Childhood Neglect and Abuse: Corroboration in a Series of Sisters
Bifulco, A., Brown, G. W., Lillie, A., & Jarvis, J. (1997) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1997.tb01520.x · Cited by 251
Li, M., Gao, Q., & Yu, T. (2023) · BMC Cancer · doi:10.1186/s12885-023-11325-z · Cited by 187
“However, weighted Kappa offers several advantages [15].”
Reiss, D. (1995) · Archives of General Psychiatry · doi:10.1001/archpsyc.1995.03950230039007 · Cited by 252
Explainability as a non-functional requirement: challenges and recommendations
Chazette, L., & Schneider, K. (2020) · Requirements Engineering · doi:10.1007/s00766-020-00333-1 · Cited by 220
“The degree of agreement was calculated using the Cohen’s kappa statistic [23].”
MethodWhen Group Work Doesn’t Work: Insights from Students
Chang, Y., & Brickman, P. (2018) · CBE—Life Sciences Education · doi:10.1187/cbe.17-09-0199 · Cited by 206
“Interrater reliability (Cohen, 1968) between the two authors was achieved at 0.”
BackgroundMODALITY-SPECIFIC AND SUPRAMODAL MECHANISMS OF APRAXIA
DE RENZI, E., FAGLIONI, P., & SORGATO, P. (1982) · Brain · doi:10.1093/brain/105.2.301 · Cited by 247
Communication in Critical Care Environments: Mobile Telephones Improve Patient Care
Soto, R. G., Chu, L. F., Goldman, J. M., Rampil, I. J., & Ruskin, K. J. (2006) · Anesthesia & Analgesia · doi:10.1213/01.ane.0000194506.79408.79 · Cited by 251
“Testretest reliability was assessed using unweighted (2 2 contingency tables) Cohen’s Kappa statistic with 95% confidence intervals (CIs) ( 2 ).”
MethodIssues in Comparisons Between Meta-analyses and Large Trials
Ioannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.14.1089 · Cited by 253
(1990) · American Journal of Psychiatry · doi:10.1176/ajp.147.1.76 · Cited by 244
Definitions of schizophrenia: concordance and prediction of outcome
Brockington, I. F., Kendell, R. E., & Leff, J. P. (1978) · Psychological Medicine · doi:10.1017/s0033291700016068 · Cited by 248
Conceptual problem solving in high school physics
Docktor, J. L., Strand, N. E., Mestre, J. P., & Ross, B. H. (2015) · Physical Review Special Topics - Physics Education Research · doi:10.1103/physrevstper.11.020106 · Cited by 206
“02 indicating substantial agreement [36].”
Suh, M. H., Zangwill, L. M., Manalastas, P. I. C., Belghith, A., Yarmohammadi, A., Medeiros, F. A., Diniz-Filho, A., Saunders, L. J., & Weinreb, R. N. (2016) · Ophthalmology · doi:10.1016/j.ophtha.2016.09.002 · Cited by 248
“The k coefficient was calculated to evaluate interobserver agreement in determining the parapapillary deep-layer microvasculature dropout and focal LC defects.(39,40) Interobserver reproducibility of the measurement of the bPPA area was measured by calculating the intraclass correlation coefficients of all included pa…”
MethodCriterion-Referenced Testing and Measurement: A Review of Technical Issues and Developments
Hambleton, R. K., Swaminathan, H., Algina, J., & Coulson, D. B. (1978) · Review of Educational Research · doi:10.2307/1169908 · Cited by 242
“The proper t ies of K have been discussed in detai l by Cohen (1960,1968) and Fleiss , Cohen, and E v e r i t t (1969) as well as by o thers .”
MethodChang, W., Ma, J., Chiu, H., Lin, K., & Lee, P. (2009) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2009.05085.x · Cited by 226
Comparison of SARS-CoV-2 detection in nasopharyngeal swab and saliva
Iwasaki, S., Fujisawa, S., Nakakubo, S., Kamada, K., Yamashita, Y., Fukumoto, T., Sato, K., Oguri, S., Taki, K., Senjo, H., et al. (2020) · Journal of Infection · doi:10.1016/j.jinf.2020.05.071 · Cited by 240
Fonner, K. L., & Stache, L. C. (2012) · New Technology, Work and Employment · doi:10.1111/j.1468-005x.2012.00290.x · Cited by 201
Duplex ultrasound scanning in the diagnosis of renal artery stenosis: A prospective evaluation
Taylor, D. C., Kettler, M. D., Moneta, G. L., Kohler, T. R., Kazmers, A., Beach, K. W., & Strandness, D. E. (1988) · Journal of Vascular Surgery · doi:10.1016/0741-5214(88)90156-5 · Cited by 248
Cicchetti, D., Bronen, R., Spencer, S., Haut, S., Berg, A., Oliver, P., & Tyrer, P. (2006) · Journal of Nervous & Mental Disease · doi:10.1097/01.nmd.0000230392.83607.c5 · Cited by 241
“As an early example, Coombs (1953) echoed the words of Stevens when he stated: “This level of measurement is so primitive that it is not always characterized as a measurement, but it is a necessary condition for all higher levels of measurement” (p. 473). More than 4 decades later, Grimm and Yarnold (1995), in agreeme…”
BackgroundInfluentialSample-size calculations for Cohen's kappa.
Cantor, A. B. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.150 · Cited by 240
“Weighted kappa allows different types of disagreement to have differing weights ( Cohen, 1968 ).”
BackgroundLauritsen, M. B., Jørgensen, M., Madsen, K. M., Lemcke, S., Toft, S., Grove, J., Schendel, D. E., & Thorsen, P. (2010) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-009-0818-0 · Cited by 239
The role of measurement reliability in clinical trials
Lachin, J. M. (2004) · Clinical Trials · doi:10.1191/1740774504cn057oa · Cited by 234
“While originally developed for application to a binary variable, Kappa was later generalized by Cohen [26] to apply to a categorical variable and to ordinal variables, the latter termed weighted K.”
MethodSteering of westerly storms over western North America at the Last Glacial Maximum
Oster, J. L., Ibarra, D. E., Winnick, M. J., & Maher, K. (2015) · Nature Geoscience · doi:10.1038/ngeo2365 · Cited by 237
de Ru, M. H., Boelens, J. J., Das, A. M., Jones, S. A., van der Lee, J. H., Mahlaoui, N., Mengel, E., Offringa, M., O'Meara, A., Parini, R., et al. (2011) · Orphanet Journal of Rare Diseases · doi:10.1186/1750-1172-6-55 · Cited by 236
“Intra-observer reproducibility was quantified by Cohen’s kappa [22].”
McGuffin, P. (1984) · Archives of General Psychiatry · doi:10.1001/archpsyc.1984.01790170015002 · Cited by 239
Amicus Curiae and the Role of Information at the Supreme Court
Spriggs, J. F., & Wahlbeck, P. J. (1997) · Political Research Quarterly · doi:10.1177/106591299705000206 · Cited by 231
Extension of the Kappa Coefficient
Kraemer, H. C. (1980) · Biometrics · doi:10.2307/2529972 · Cited by 237
Ducharme, F. M., Chalut, D., Plotnick, L., Savdie, C., Kudirka, D., Zhang, X., Meng, L., & McGillivray, D. (2008) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2007.08.034 · Cited by 233
Automatic Assessment of Depression Based on Visual Cues: A Systematic Review
Pampouchidou, A., Simos, P. G., Marias, K., Meriaudeau, F., Yang, F., Pediaditis, M., & Tsiknakis, M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2724035 · Cited by 228
“permit direct comparisons between classification approaches, Cohen’s Kappa statistic [169], a chance and”
MethodResolving "unfinished business": Efficacy of experiential therapy using empty-chair dialogue.
Paivio, S. C., & Greenberg, L. S. (1995) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.63.3.419 · Cited by 233
Inequalities between multi-rater kappas
Warrens, M. J. (2010) · Advances in Data Analysis and Classification · doi:10.1007/s11634-010-0073-4 · Cited by 226
Better Access, Quality, and Cost for Clinically Complex Veterans with Home‐Based Primary Care
Edes, T., Kinosian, B., Vuckovic, N. H., Olivia Nichols, L., Mary Becker, M., & Hossain, M. (2014) · Journal of the American Geriatrics Society · doi:10.1111/jgs.13030 · Cited by 229
Detecting and Monitoring Hate Speech in Twitter
Pereira-Kohatsu, J. C., Quijano-Sánchez, L., Liberatore, F., & Camacho-Collados, M. (2019) · Sensors · doi:10.3390/s19214654 · Cited by 214
“The two main statistical measures of the agreement percentage, popular in labeling processes and areas such as psychology and education [76], are Cohens’ kappa [77], which measures the degree of agreement between two raters, and Fliess’ kappa [78], which makes it possible to measure the concordance among two or more r…”
BackgroundDNA methylation-based predictors of health: applications and statistical considerations
Yousefi, P. D., Suderman, M., Langdon, R., Whitehurst, O., Davey Smith, G., & Relton, C. L. (2022) · Nature Reviews Genetics · doi:10.1038/s41576-022-00465-w · Cited by 235
The development of a valid and reliable scale for rating anxiety in dementia (RAID)
SHANKAR, K. K., WALKER, M., FROST, D., & ORRELL, M. W. (1999) · Aging & Mental Health · doi:10.1080/13607869956424 · Cited by 216
Decentring and distraction reduce overgeneral autobiographical memory in depression
WATKINS, E., TEASDALE, J. D., & WILLIAMS, R. M. (2000) · Psychological Medicine · doi:10.1017/s0033291799002263 · Cited by 231
“rater agreement on a random sample of 8% of memories (N ̄ 210) was, respectively, 93±8% and 86±4%, κ ̄ 0±86 and κ ̄ 0±72 (Cohen, 1968) controlling for agreement due to chance).”
BackgroundItem Analysis of the Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST)
Demers, L., Weiss-Lambrou, R., & Ska, B. (2000) · Assistive Technology · doi:10.1080/10400435.2000.10132015 · Cited by 220
Measuring agreement in medical informatics reliability studies
Hripcsak, G., & Heitjan, D. F. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(02)00500-2 · Cited by 226
“A weighted version of kappa has also been defined [10,16] which allows one to assign different penalties to different mismatches.”
BackgroundSleep scoring using artificial neural networks
Ronzhina, M., Janoušek, O., Kolářová, J., Nováková, M., Honzík, P., & Provazník, I. (2012) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2011.06.003 · Cited by 229
Savelsbergh, E. R., Prins, G. T., Rietbergen, C., Fechner, S., Vaessen, B. E., Draijer, J. M., & Bakker, A. (2016) · Educational Research Review · doi:10.1016/j.edurev.2016.07.003 · Cited by 188
“Interrater agreement, as expressed in Cohen's Kappa (Cohen, 1968) was found acceptable (Table 2).”
O'Donoghue, M. F., Duncan, J. S., & Sander, J. W. A. S. (1996) · Epilepsia · doi:10.1111/j.1528-1157.1996.tb00610.x · Cited by 223
Child and maternal contributions to shared reading: Effects on language and literacy development
Deckner, D. F., Adamson, L. B., & Bakeman, R. (2006) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2005.12.001 · Cited by 205
Savant skills in autism: psychometric approaches and parental reports
Howlin, P., Goode, S., Hutton, J., & Rutter, M. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0328 · Cited by 198
“Inter-rater reliability was calculated using weighted kappas (Cohen 1968).”
MethodKelley, G. A., Kelley, K. S., & Hootman, J. M. (2015) · Arthritis Research & Therapy · doi:10.1186/s13075-015-0533-5 · Cited by 92
Brondani, J. T., Mama, K. R., Luna, S. P. L., Wright, B. D., Niyom, S., Ambrosio, J., Vogel, P. R., & Padovani, C. R. (2013) · BMC Veterinary Research · doi:10.1186/1746-6148-9-143 · Cited by 213
“The agreement between each blinded observer and the ‘gold standard’ was determined by the weighted Kappa coefficient [45].”
MethodKowalewski, M., Pawliszak, W., Malvindi, P. G., Bokszanski, M. P., Perlinski, D., Raffa, G. M., Kowalkowska, M. E., Zaborowska, K., Navarese, E. P., Kolodziejczak, M., et al. (2016) · The Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2015.08.042 · Cited by 225
Kelley, G. A., Kelley, K. S., & Tran, Z. V. (2004) · Journal of Women's Health · doi:10.1089/jwh.2004.13.1148 · Cited by 225
Bech, P., Timmerby, N., Martiny, K., Lunde, M., & Soendergaard, S. (2015) · BMC Psychiatry · doi:10.1186/s12888-015-0529-3 · Cited by 216
“[25].”
Westen, D., Lohr, N., Silk, K. R., & Gold, L. (1990) · Psychological Assessment · doi:10.1037//1040-3590.2.4.355 · Cited by 217
Sympathy, distress, and prosocial behavior of preschool children in four cultures
Trommsdorff, G., Friedlmeier, W., & Mayer, B. (2007) · International Journal of Behavioral Development · doi:10.1177/0165025407076441 · Cited by 210
“Additionally, weighted kappa (Cohen, 1968) was calculated to check the extent of agreement between raters.”
BackgroundVerbal analysis of doctor-patient communication
Meeuwesen, L., Schaap, C., & van der Staak, C. (1991) · Social Science & Medicine · doi:10.1016/0277-9536(91)90091-p · Cited by 229
Gibson, R. S., & Ferguson, E. (2008) · Cited by 223
Disability, race/ethnicity and gender: themes of cultural oppression, acts of individual resistance
McDonald, K. E., Keys, C. B., & Balcazar, F. E. (2007) · American Journal of Community Psychology · doi:10.1007/s10464-007-9094-3 · Cited by 215
“To capture inter-rater agreement beyond chance, we calculated Cohen’s Kappa which, at 88% was highly satisfactory (Cohen, 1968). The first author also recoded 20% of the interviews two months after coding was finalized to assess temporal reliability (Foster-Fishman & Keys, 1997). No significant discrepancies were foun…”
MethodSorror, M. L., Logan, B. R., Zhu, X., Rizzo, J. D., Cooke, K. R., McCarthy, P. L., Ho, V. T., Horowitz, M. M., & Pasquini, M. C. (2015) · Biology of Blood and Marrow Transplantation · doi:10.1016/j.bbmt.2015.04.004 · Cited by 217
Anderson, L. G., Rocliffe, S., Haddaway, N. R., & Dunn, A. M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0140833 · Cited by 201
“Titles and abstracts were vetted by two reviewers (LA and SR) and the Kappa statistic [46] was used to evaluate inter-reviewer agreement (Kappa = 0.”
MethodStoolmiller, M., Eddy, J. M., & Reid, J. B. (2000) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.68.2.296 · Cited by 214
“Various statistics, for example, Cohen's kappa (Cohen, 1968), can be computed to assess how well observers agree under the coding system, and if agreement is high, the code and the observer training are judged adequate.”
BackgroundThe Genodisc Consortium, Jamaludin, A., Lootus, M., Kadir, T., Zisserman, A., Urban, J., Battié, M. C., Fairbank, J., & McCall, I. (2017) · European Spine Journal · doi:10.1007/s00586-017-4956-3 · Cited by 211
Broger, T., Sossen, B., du Toit, E., Kerkhoff, A. D., Schutz, C., Ivanova Reipold, E., Ward, A., Barr, D. A., Macé, A., Trollip, A., et al. (2019) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(19)30001-5 · Cited by 218
Mason, A., Rioux, J., Clarke, S. E., Costa, A., Schmidt, M., Keough, V., Huynh, T., & Beyea, S. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2930338 · Cited by 204
“The weighted kappa [35] between radiologists who rated the same set of images is presented in Table III.”
BackgroundHow I assess comorbidities before hematopoietic cell transplantation
Sorror, M. L. (2013) · Blood · doi:10.1182/blood-2012-09-455063 · Cited by 217
Value of the Electroencephalogram in Adult Patients With Untreated Idiopathic First Seizures
van Donselaar, C. A., Schimsheimer, R.-J., Geerts, A. T., & Declerck, A. C. (1992) · Archives of Neurology · doi:10.1001/archneur.1992.00530270045017 · Cited by 219
Similarities between Friends and Nonfriends in Middle Childhood
Haselager, G. J. T., Hartup, W. W., van Lieshout, C. F. M., & Riksen-Walraven, J. M. A. (1998) · Child Development · doi:10.1111/j.1467-8624.1998.tb06167.x · Cited by 212
“Owing to measurement restrictions (see below), similarities between friends and nonfriends in the nominations they gave to their classmates are assessed differently-by means of the kappa statistic (Cohen, 1968).”
BackgroundMethodThe effect of sea level on glacial Indo-Pacific climate
DiNezio, P. N., & Tierney, J. E. (2013) · Nature Geoscience · doi:10.1038/ngeo1823 · Cited by 211
Long-term land cover dynamics by multi-temporal classification across the Landsat-5 record
Sexton, J. O., Urban, D. L., Donohue, M. J., & Song, C. (2013) · Remote Sensing of Environment · doi:10.1016/j.rse.2012.10.010 · Cited by 211
“This is accomplished by defining the weights used to compute κw as vector w of class weights, with each elementwj equal to the ratio of the population proportion to the sample (reference) proportion for each class (Stehman & Czaplewski, 1998): wj ¼ Nj N nj n ¼ n Nj N nj ; ðA12Þ i.e., wj ¼ Pj pj : ðA13Þ The standardize…”
BackgroundInfluentialEna, J., & Valls, V. (2005) · Clinical Infectious Diseases · doi:10.1086/427802 · Cited by 217
“We used the k statistic to measure the chance-corrected agreement between the 2 independent reviewers with regard to trial inclusion and quality assessment [7].”
MethodThe impact of green experience on customer satisfaction: evidence from TripAdvisor
Yu, Y., Li, X., & Jai, T.-M. (. (2017) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-07-2015-0371 · Cited by 196
Burgio, L. D., Allen-Burge, R., Roth, D. L., Bourgeois, M. S., Dijkstra, K., Gerstle, J., Jackson, E., & Bankester, L. (2001) · The Gerontologist · doi:10.1093/geront/41.4.449 · Cited by 212
“Interobserver agreement was calculated using Cohen’s kappa (Cohen, 1968; Hays, 1994).”
MethodDocktor, J. L., Dornfeld, J., Frodermann, E., Heller, K., Hsu, L., Jackson, K. A., Mason, A., Ryan, Q. X., & Yang, J. (2016) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.12.010130 · Cited by 182
“One method for quantitatively judging reliability is through the use of the weighted kappa [61].”
MethodKlouwenberg, P. M. C. K., Ong, D. S. Y., Bos, L. D. J., de Beer, F. M., van Hooijdonk, R. T. M., Huson, M. A., Straat, M., van Vught, L. A., Wieske, L., Horn, J., et al. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e3182923712 · Cited by 218
“We calculated Cohen’s kappa (κ) for categorical variables (source of infection and causative pathogen) and a weighted κ for ordinal variables (plausibility of infection) (15).”
MethodThe CAST (Childhood Asperger Syndrome Test): Test accuracy
Williams, J., Scott, F., Stott, C., Allison, C., Bolton, P., Baron-Cohen, S., & Brayne, C. (2005) · Autism · doi:10.1177/1362361305049029 · Cited by 208
Empirical Methods for Bioethics: A Primer
(2007) · Advances in Bioethics · doi:10.1016/s1479-3709(2007)11 · Cited by 208
Sullivan, P. M., & Knutson, J. F. (1998) · Child Abuse & Neglect · doi:10.1016/s0145-2134(97)00175-0 · Cited by 213
Statistical methods for assessing agreement for ordinal data
Jakobsson, U., & Westergren, A. (2005) · Scandinavian Journal of Caring Sciences · doi:10.1111/j.1471-6712.2005.00368.x · Cited by 203
The Comprehensive System for the Rorschach: A Critical Examination
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996) · Psychological Science · doi:10.1111/j.1467-9280.1996.tb00658.x · Cited by 211
“However, percentage of agreement has long been recognized as a potentially inadequate and misleading measure of reliability (Cohen, 1960,1968; Fleiss, 1981; Light, 1971; see also Jensen, 1965), The problem is that percentage of agreement makes no adjustment for agreement by chance and can therefore…”
BackgroundKeller, B. M., Nathan, D. L., Wang, Y., Zheng, Y., Gee, J. C., Conant, E. F., & Kontos, D. (2012) · Medical Physics · doi:10.1118/1.4736530 · Cited by 209
Kennedy, C. E., Fonner, V. A., Armstrong, K. A., Denison, J. A., Yeh, P. T., O’Reilly, K. R., & Sweat, M. D. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0925-0 · Cited by 212
“Cohen’s kappa (κ) [31] was estimated for individual bias tool items and weighted kappa (κw) [32] for the count of endorsed items (the sum of individual item responses).”
MethodPesaresi, S., Biondi, E., & Casavecchia, S. (2017) · Journal of Maps · doi:10.1080/17445647.2017.1413017 · Cited by 205
“To determine the validity of these maps also at a regional scale for the Ombrotypes and Thermotypes according to the weighed Cohen’s K statistic (Cohen, 1968), we assessed the degree of agreement between the high-resolution maps constructed for the Sardinia region (Canu et al., 2015) (see…”
MethodMartinez-Calderon, J., Jensen, M. P., Morales-Asencio, J. M., & Luque-Suarez, A. (2019) · The Clinical Journal of Pain · doi:10.1097/ajp.0000000000000676 · Cited by 188
State of the art of cyber-physical systems security: An automatic control perspective
Zacchia Lun, Y., D’Innocenzo, A., Smarra, F., Malavolta, I., & Di Benedetto, M. D. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2018.12.006 · Cited by 211
“[57], two researchers assessed a random sample of the studies and the inter-researcher agreement has been measured using the Cohen Kappa statistic [63].”
A boost of positive affect: The perks of sharing positive experiences
Lambert, N. M., Gwinn, A. M., Baumeister, R. F., Strachman, A., Washburn, I. J., Gable, S. L., & Fincham, F. D. (2013) · Journal of Social and Personal Relationships · doi:10.1177/0265407512449400 · Cited by 187
“Weighted kappa coefficients (Cohen, 1968) were used to determine inter-rater reliability for the three positive emotions, including love ( w ¼ .49), appreciation ( w ¼ .33), and happiness ( w ¼ .47).”
MethodSentiment Polarity Detection for Software Development
Calefato, F., Lanubile, F., Maiorano, F., & Novielli, N. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9546-9 · Cited by 171
“As an evidence of the reliability of the coding schema and procedure, we computed the weighted Cohen ’ s Kappa among the pairs of raters (Cohen 1968).”
MethodPossible role ofHelicobacter pylori infection in early gastric cancer development
Asaka, M., Kimura, T., Kato, M., Kudo, M., Miki, K., Ogoshi, K., Kato, T., Tatsuta, M., & Graham, D. Y. (1994) · Cancer · doi:10.1002/1097-0142(19940601)73:11<2691::aid-cncr2820731107>3.0.co;2-2 · Cited by 213
Translation of questionnaires and issues of equivalence
Chang, A. M., Chau, J. P. C., & Holroyd, E. (1999) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.1999.00891.x · Cited by 209
Kopp, B., Kunkel, A., Flor, H., Platz, T., Rose, U., Mauritz, K.-H., Gresser, K., McCulloch, K. L., & Taub, E. (1997) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(97)90427-5 · Cited by 214
Kirişli, H. A., Schaap, M., Metz, C. T., Dharampal, A. S., Meijboom, W. B., Papadopoulou, S. L., Dedic, A., Nieman, K., de Graaf, M. A., Meijs, M. F. L., et al. (2013) · Medical Image Analysis · doi:10.1016/j.media.2013.05.007 · Cited by 205
“Therefore, we use the linearly weighted Cohen’s Kappa metric (Cohen, 1968).”
Neilan, T. G., Coelho-Filho, O. R., Danik, S. B., Shah, R. V., Dodson, J. A., Verdini, D. J., Tokuda, M., Daly, C. A., Tedrow, U. B., Stevenson, W. G., et al. (2013) · JACC: Cardiovascular Imaging · doi:10.1016/j.jcmg.2013.05.013 · Cited by 205
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207
“For example, weighted κ (Cohen, 1968) extends Cohen’s κ by adding weights to each possible pair of score assignments.”
BackgroundTaylor, M. R., & Lalani, N. (2013) · Academic Emergency Medicine · doi:10.1111/acem.12150 · Cited by 205
Expert system verification and validation: a survey and tutorial
O'Keefe, R. M., & O'Leary, D. E. (1993) · Artificial Intelligence Review · doi:10.1007/bf00849196 · Cited by 205
“The weighted kappa (Cohen 1968) measures overall agreement across all categories, and was used by Hickham et al. (1985) in the ONCOCIN validation.”
MethodThe Impact of Maternal Characteristics and Contextual Variables on Infant–Mother Attachment
Huth-Bocks, A. C., Levendosky, A. A., Bogat, G. A., & Von Eye, A. (2004) · Child Development · doi:10.1111/j.1467-8624.2004.00688.x · Cited by 186
Blair, A., Tarone, R., Sandler, D., Lynch, C. F., Rowland, A., Wintersteen, W., Steen, W. C., Samanic, C., Dosemeci, M., & Alavanja, M. C. R. (2002) · Epidemiology · doi:10.1097/00001648-200201000-00015 · Cited by 198
Automated Classification of Radiographic Knee Osteoarthritis Severity Using Deep Neural Networks
Thomas, K. A., Kidziński, Ł., Halilaj, E., Fleming, S. L., Venkataraman, G. R., Oei, E. H. G., Gold, G. E., & Delp, S. L. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190065 · Cited by 192
“Development of a region proposal model (13,15) that combines automatic joint space identification with KL classification in a single model has been evaluated on portions of the OAI dataset (16), but this approach requires the initial manual annotation of a training set of images.”
BackgroundMethodNocturnal hypoxaemia and quality of sleep in patients with chronic obstructive lung disease.
Cormick, W., Olson, L. G., Hensley, M. J., & Saunders, N. A. (1986) · Thorax · doi:10.1136/thx.41.11.846 · Cited by 202
Financial analysts' reports: an extended institutional theory evaluation
Fogarty, T. J., & Rogers, R. K. (2005) · Accounting, Organizations and Society · doi:10.1016/j.aos.2004.06.003 · Cited by 197
Joint Task Force of the EFNS and the PNS (2010) · Journal of the Peripheral Nervous System · doi:10.1111/j.1529-8027.2010.00269.x · Cited by 197
Minunno, R., O'Grady, T., Morrison, G. M., & Gruner, R. L. (2021) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.110935 · Cited by 185
“The resulting Cohen ’ s kappa is 0.86, representing a high agreement between the authors (see Appendix C; Cohen [34]).”
ResultInfluential
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.