Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 901–1,000 of 1,098
Mental terms in mothers' and children's speech: similarities and relationships
Furrow, D. R., Moore, C. L., Davidge, J., & Chiasson, L. A. (1992) · Journal of Child Language · doi:10.1017/s0305000900011594 · Cited by 103
A new approach to categorising continuous variables in prediction models: Proposal and validation
Barrio, I., Aróstegui, I., Rodríguez‐Álvarez, M. X., & Quintana, J. M. (2015) · Statistical Methods in Medical Research · doi:10.1177/0962280215601873 · Cited by 111
Sánchez‐López, M., Martínez‐Vizcaíno, V., García‐Hermoso, A., Jiménez‐Pavón, D., & Ortega, F. B. (2014) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/sms.12267 · Cited by 106
“The test–retest reliability of the IFIS was examined by quadratic weighted Kappa coefficient, (Cohen, 1968).”
MethodA benchmark study on sentiment analysis for software engineering research
Novielli, N., Girardi, D., & Lanubile, F. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196403 · Cited by 68
“Furthermore, we report the weighted kappa ( (cid:4) ) by Cohen [9], a measure of interrater agreement that we use to assess both the agreement with gold labels (RQ1) and the agreement among the three tools (RQ2).”
MethodDetecting certainness in spoken tutorial dialogues
Liscombe, J., Hirschberg, J., & Venditti, J. J. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-581 · Cited by 99
“Inter-labeler agreement for this annotation was calculated using Cohen’s Kappa statistic [7] on a subset of the data consisting of 505 student turns labeled for certainness by three different labelers.”
MethodExaminer alignment and assessment in clinical periodontal research
Hefti, A. F., & Preshaw, P. M. (2012) · Periodontology 2000 · doi:10.1111/j.1600-0757.2011.00436.x · Cited by 97
Nicholson, A. G., Perry, L. J. P., Cury, P. M., Jackson, P., MCCORMICK, C., CORRIN, B., & Wells, A. U. (2001) · Histopathology · doi:10.1046/j.1365-2559.2001.01078.x · Cited by 109
Dewey, M. E. (1983) · British Journal of Psychiatry · doi:10.1192/bjp.143.5.487 · Cited by 100
Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis.
Orwin, R. G., & Cordray, D. S. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.97.1.134 · Cited by 102
Application of the Suchey-Brooks method to three-dimensional imaging of the pubic symphysis
Telmon, N., Gaston, A., Chemla, P., Blanc, A., Joffre, F., & Rougé, D. (2005) · Journal of Forensic Sciences · doi:10.1520/jfs2004326 · Cited by 101
Determinants of dental care visits among low-income African-American children
Sohn, W., Ismail, A. I., Amaya, A., & Lepkowski, J. M. (2007) · The Journal of the American Dental Association · doi:10.14219/jada.archive.2007.0163 · Cited by 114
Learning from avatars: Learning assistants practice physics pedagogy in a classroom simulator
Chini, J. J., Straub, C. L., & Thomas, K. H. (2016) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.12.010117 · Cited by 106
“We used cycles of open, axial and selective coding, and constant comparison to identify pedagogical strategies and to link evidence from the videos with evidence from the written reflections [36,37].”
MethodReliability of pubertal self‐assessment in Hong Kong Chinese children
Chan, N. P., Sung, R. Y., Kong, A. P., Goggins, W. B., So, H. K., & Nelson, E. A. S. (2008) · Journal of Paediatrics and Child Health · doi:10.1111/j.1440-1754.2008.01311.x · Cited by 102
An improved Fuzzy Kappa statistic that accounts for spatial autocorrelation
Hagen‐Zanker, A. (2009) · International Journal of Geographical Information Systems · doi:10.1080/13658810802570317 · Cited by 101
“weighted Kappa (Cohen 1968).”
BackgroundAssessing transcription agreement: Methodological aspects
Catia Cucchiarini, A. (1996) · Clinical Linguistics & Phonetics · doi:10.3109/02699209608985167 · Cited by 101
Vuille‐Lessard, É., Lebouché, B., Lennox, L., Routy, J., Costiniuk, C. T., Pexos, C., Giannakis, A., Szabo, J., Klein, M. B., & Sebastiani, G. (2016) · AIDS · doi:10.1097/qad.0000000000001241 · Cited by 107
The Behavioral Dynamics of Mutual Responsiveness in Early Face-to-Face Mother-Infant Interactions
Symons, D. K., & Moran, G. (1987) · Child Development · doi:10.1111/j.1467-8624.1987.tb03861.x · Cited by 102
Life events, symptoms and social support
Miller, P. M., Ingham, J. G., & Davidson, S. (1976) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(76)90052-0 · Cited by 101
Applying the behaviour change technique (BCT) taxonomy v1: A study of coder training
Wood, C. E., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., & Michie, S. (2015) · Translational Behavioral Medicine · doi:10.1007/s13142-014-0290-z · Cited by 100
Kelley, G. A., & Kelley, K. S. (2015) · PLoS ONE · doi:10.1371/journal.pone.0129181 · Cited by 109
Airas, M., & Alku, P. (2006) · Phonetica · doi:10.1159/000091405 · Cited by 117
Madaras-Kelly, K., Jones, M., Remington, R., Hill, N., Huttner, B., & Samore, M. (2014) · Infection Control & Hospital Epidemiology · doi:10.1086/677633 · Cited by 99
An attributional analysis of expressed emotion in Mexican-American families with schizophrenia.
Weisman, A., López, S. R., Karno, M., & Jenkins, J. (1993) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.102.4.601 · Cited by 99
Grouven, U., Bender, R., Ziegler, A., & Lange, S. (2007) · DMW - Deutsche Medizinische Wochenschrift · doi:10.1055/s-2007-959046 · Cited by 100
Biliary dysplasia as a marker of cholangiocarcinoma in primary sclerosing cholangitis
Fleming, K. A., Boberg, K. M., Glaumann, H., Bergquist, A. M., Smith, D., & Clausen, O. P. F. (2001) · Journal of Hepatology · doi:10.1016/s0168-8278(00)00034-9 · Cited by 100
“Inter-observer agreement for the 49 slides between the two rounds of scorings was evaluated using the k coef®cient of Cohen [12] as calculated by the computer package Stata (1997).”
MethodBehavioral Observation and Coding
Heyman, R. E., Lorber, M. F., Eddy, J. M., & West, T. V. (2014) · Handbook of Research Methods in Social and Personality Psychology · doi:10.1017/cbo9780511996481.018 · Cited by 87
Bell, E. H., Pugh, S. L., McElroy, J. P., Gilbert, M. R., Mehta, M. P., Klimowicz, A. C., Magliocco, A. M., Bredel, M., Robe, P. A. J. T., Grosu, A., et al. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2016.6020 · Cited by 107
Evaluating Forecasting Methods by Considering Different Accuracy Measures
Mehdiyev, N., Enke, D. L., Fettke, P., & Loos, P. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.09.332 · Cited by 119
Ophthalmoscopy versus fundus photographs for detecting and grading diabetic retinopathy.
Kinyoun, J. L., Martin, D., Fujimoto, W. Y., & Leonetti, D. L. (1992) · PubMed · Cited by 99
Daskalakis, Z. J., Molnar, G. F., Christensen, B. K., Sailer, A., Fitzgerald, P. B., & Chen, R. E. (2003) · Clinical Neurophysiology · doi:10.1016/s1388-2457(03)00038-5 · Cited by 99
Ames, D. (1990) · British Journal of Psychiatry · doi:10.1192/bjp.156.5.667 · Cited by 97
“Psychiatric diagnoses The test of the author's reliabilityin using the GMS gave a mean weightedkappa (Cohen, 1968)for the 184items comprising the 21 subscales of the GMS described by Gurland et a!”
MethodTherapist-delivered and self-help interventions for gambling problems: A review of contents
Rodda, S. N., Merkouris, S., Abraham, C., Hodgins, D. C., Cowlishaw, S., & Dowling, N. A. (2018) · Journal of Behavioral Addictions · doi:10.1556/2006.7.2018.44 · Cited by 103
Aoyama, H., Uchida, K., Aoyama, K., Pechlivanoglou, P., Englesakis, M. F., Yamada, Y., & Fan, E. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.8116 · Cited by 106
von Berlepsch, D., Lemke, F., & Gorton, M. (2022) · Journal of Business Ethics · doi:10.1007/s10551-022-05268-x · Cited by 120
“Agreement by chance is eliminated by a Cohen’s Kappa of 91.3%, which exceeds substantially the recommended threshold (88.4%), as suggested in the literature (Cohen, 1968; Lombard et al., 2002; Perreault Jr & Leigh, 1989).”
Influentialde Moura, I. R., Teles, A. S., Viana, D., Marques, J., Coutinho, L., & Silva, F. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104278 · Cited by 109
Bangdiwala, S. I., & Shankar, V. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-97 · Cited by 104
Mota, P., Pascoal, A. G., Sancho, F., Carita, A. I., & Bø, K. (2013) · Manual Therapy · doi:10.1016/j.math.2012.10.013 · Cited by 98
“The level of agreement on palpation between the two days of measurements (intra-rater) and between the two testers (interrater) was determined by means of Spearman’s rho and weighted Kappa (Cohen, 1968).”
MethodInfluentialThe prevalence of disabilities and maltreatment among runaway children
Sullivan, P. M., & Knutson, J. F. (2000) · Child Abuse & Neglect · doi:10.1016/s0145-2134(00)00181-2 · Cited by 98
The Composition of the Negative Syndrome of Chronic Schizophrenia
Kibel, D. A., Laffont, I., & Liddle, P. F. (1993) · British Journal of Psychiatry · doi:10.1192/bjp.162.6.744 · Cited by 99
“Inter-rater reliabilityfor individualitemswascalculated using Cohen's kappa (Cohen, 1968), and Pearson correlation wasusedforthetotalscalescores.”
MethodEffectiveness of Amiodarone for Conversion of Atrial Fibrillation to Sinus Rhythm: A Meta-analysis
Letelier, L. M., Udol, K., Ena, J., Weaver, B., & Guyatt, G. H. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.7.777 · Cited by 97
Personifying Country of Origin Research
Nebenzahl, I. D., Jaffe, E. D., & Usunier, J. (2003) · IRIS · Cited by 97
Bekchanova, M., Campion, L., Bruns, S. B., Kuppens, T. E., Lehmann, J. C., Jozefczak, M., Cuypers, A., & Malina, R. M. (2024) · Environmental Evidence · doi:10.1186/s13750-024-00326-5 · Cited by 114
“Google Scholar returned many results that were related and unrelated to the review objective; therefore, following [46] and [47], we only downloaded the first 1000 relevant results using Publish or Perish software, as stated in the protocol.”
MethodDual branch deep learning network for detection and stage grading of diabetic retinopathy
Shakibania, H., Raoufi, S., Pourafkham, B., Khotanlou, H., & Mansoorizadeh, M. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106168 · Cited by 114
“Higher scores indicate better agreement between the raters or better model performance [51].”
BackgroundTelenius, E. W., Engedal, K., & Bergland, A. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008321 · Cited by 106
“The weighted κ score measures the agreement among raters, adjusted for the amount of agreement expected by chance and the magnitude of disagreement.(47) A κ value of 0.”
BackgroundIntensive Care Unit Admitting Patterns in the Veterans Affairs Health Care System
Chen, L. M., Render, M. L., Sales, A. E., Kennedy, E. H., Wiitala, W. L., & Hofer, T. P. (2012) · Archives of Internal Medicine · doi:10.1001/archinternmed.2012.2606 · Cited by 101
AN EXPLORATION OF FEEDING DIFFICULTIES IN CHILDREN WITH DOWN SYNDROME
Spender, Q. W., Stein, A. L., Dennis, J., Reilly, S., Percy, E., & Cave, D. (1996) · Developmental Medicine & Child Neurology · doi:10.1111/j.1469-8749.1996.tb12138.x · Cited by 108
Day, T. L., Farnell, S., Haynes, S. C., Wainwright, S. P., & Wilson-Barnett, J. (2002) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2002.02240.x · Cited by 97
Ozone therapy for the treatment of dental caries
Rickard, G. D., Richardson, R. J., Johnson, T. M., McColl, D. C., & Hooper, L. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004153.pub2 · Cited by 97
RetractedDementia and Cognitive Impairment in the Oldest Old in the Community Prevalence and Comorbidity
Fichter, M. M., Meller, I., Schröppel, H., & Steinkirchner, R. (1995) · British Journal of Psychiatry · doi:10.1192/bjp.166.5.621 · Cited by 101
“Calculations of inter-rater reliability were based on these data; for pairs of interviewers, weighted-kappa and kappa coefficients were calcu lated (Cohen, 1968).”
MethodObserver variation in the interpretation of lower limb venograms
McLachlan, M. S. F., Thomson, J., Taylor, D., Kelly, M., & Sackett, D. L. (1979) · American Journal of Roentgenology · doi:10.2214/ajr.132.2.227 · Cited by 97
Haugen, I. K., Lillegraven, S., Slatkowsky-Christensen, B., Haavardsholm, E. A., Sesseng, S., Kvien, T. K., van der Heijde, D. M. F. M., & Bøyesen, P. (2011) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2010.144527 · Cited by 109
How does tillage intensity affect soil organic carbon? A systematic review protocol
Haddaway, N. R., Hedlund, K., Jackson, L. E., Kätterer, T., Lugato, E., Thomsen, I. K., Jørgensen, H. B., & Isberg, P.-E. (2016) · Environmental Evidence · doi:10.1186/s13750-016-0052-0 · Cited by 91
“A Kappa tests [42] for consistency checking were performed to assess the level of agreement amongst members of the review team (NRH, KH and HBJ), indicating high agreement at abstract (kappa = 0.”
MethodScaria, N., Chenna, S. D., & Subramani, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64299-9_12 · Cited by 145
“For the ordinal metrics, Clear , WillYouUseIt and Bloom’sLevel , we use quadratic weighted Cohen’s κ [5] instead of simple Cohen’s κ to penalize situations with a significant rating difference.”
MethodLarge Language Model-Powered Automated Assessment: A Systematic Review
Emirtekin, E. (2025) · Applied Sciences · doi:10.3390/app15105683 · Cited by 121
Language and pragmatic functions in school‐age children on the autism spectrum
Ramberg, C., Ehlers, S., Nydén, A., Johansson, M., & Gillberg, C. (1996) · International Journal of Language & Communication Disorders · doi:10.3109/13682829609031329 · Cited by 94
Lehmann, M., Koedam, E. L. G. E., Barnes, J., Bartlett, J. W., Ryan, N. S., Pijnenburg, Y. A. L., Barkhof, F., Wattjes, M. P., Scheltens, P., & Fox, N. C. (2012) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2011.04.003 · Cited by 96
“Reliability was measured using quadratically weighted kappas (Cohen, 1968).”
Occupational Dust Exposure and Chronic Obstructive Pulmonary Disease
Muir, D. C. F. (1998) · Applied Occupational and Environmental Hygiene · doi:10.1080/1047322x.1998.10390120 · Cited by 96
Doctor‐patient communication: a study of junior house officers
Cantwell, B. M., & Ramirez, A. J. (1997) · Medical Education · doi:10.1111/j.1365-2923.1997.tb00037.x · Cited by 97
Interrater Agreement and Reliability
Goodwin, L. D. (2001) · Measurement in Physical Education and Exercise Science · doi:10.1207/s15327841mpee0501_2 · Cited by 96
“A number of different approaches or techniques are available, including simple proportions or percentages of agreement between raters; Cohen’s kappa (1960) and “weighted kappa” (Cohen, 1968); Pearson correlation coefficients, sometimes accompanied by comparisons of means (using t tests or analyses of variance [ANOVAs]…”
BackgroundMethodInfluentialMoore, P. (2008) · Journal of Clinical Psychology in Medical Settings · doi:10.1007/s10880-008-9134-7 · Cited by 96
The flashlight test and van Herick's test are poor predictors for occludable angles
Thomas, R., George, T., Braganza, A. D., & Muliyil, J. (1996) · Australian and New Zealand Journal of Ophthalmology · doi:10.1111/j.1442-9071.1996.tb01588.x · Cited by 96
Interpersonal perception and mode of resolution in marital conflict.
Knudson, R. M., Sommers, A., & Golding, S. L. (1980) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.38.5.751 · Cited by 96
Visual tracking in young infants: Evidence for object identity or object permanence?
Moore, M., Borton, R., & Darby, B. L. (1978) · Journal of Experimental Child Psychology · doi:10.1016/0022-0965(78)90076-0 · Cited by 95
In Search of a Comprehensive Value Model for Assessing Supplementary Person—Organization Fit
Clercq, S. D., Fontaine, J. R. J., & Anseel, F. (2008) · The Journal of Psychology · doi:10.3200/jrlp.142.3.277-302 · Cited by 112
“Cohen’s kappa was calculated to measure the degree of interrater agreement (Cohen, 1968).”
MethodTemperament and brain networks of attention
Posner, M. I., & Rothbart, M. K. (2018) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2017.0254 · Cited by 101
Jap, S. D., Robertson, D. C., Rindfleisch, A., & Hamilton, R. (2012) · Journal of Marketing Research · doi:10.1509/jmr.10.0121 · Cited by 103
Zhu, Y., Liu, Q., & Zhao, L. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13420-z · Cited by 140
Reliability of the Histopathologic Diagnosis of Melanocytic Dysplasia
Weinstock, M. A., Barnhill, R. L., Rhodes, A. R., & Brodsky, G. L. (1997) · Archives of Dermatology · doi:10.1001/archderm.1997.03890440019002 · Cited by 95
Demehri, S., Muhit, A. A., Zbijewski, W. B., Stayman, J. W., Yorkston, J. I., Packard, N. J., Senn, R., Yang, D., Foos, D. H., Thawait, G. K., et al. (2015) · European Radiology · doi:10.1007/s00330-014-3546-6 · Cited by 97
“The concordance in responses was analyzed using Cohen’s weighted kappa statistic [24,25] for ordinal data (i.”
Predictors of outcomes after rotator cuff repair—A meta-analysis
Raman, J., Walton, D. M., MacDermid, J. C., & Athwal, G. S. (2017) · Journal of Hand Therapy · doi:10.1016/j.jht.2016.11.002 · Cited by 95
Dahlquist, L. M., Pendley, J. S., Landtrip, D. S., Jones, C. L., & Steuber, C. P. (2002) · Health Psychology · doi:10.1037//0278-6133.21.1.94 · Cited by 94
Collaborative learning through computer-mediated argumentation
Veerman, A. L., Andriessen, J. E. B., & Kanselaar, G. (1999) · Computer-supported collaborative learning/The Computer-Supported Collaborative Learning Conference · doi:10.3115/1150240.1150317 · Cited by 95
“Inter - judge reliability of the coding system showed a Cohen's kappa (Cohen, 1968) of 0.91 for the focus variable, a kappa of information exchange categories and a kappa of 0.74 for constructive acti vities”
BackgroundInfluentialCategory Distinguishability and Observer Agreement
Darroch, J. N., & McCloud, P. (1986) · Australian Journal of Statistics · doi:10.1111/j.1467-842x.1986.tb00709.x · Cited by 99
The contextual threat of life events: the concept and its reliability
Tennant, C., Smith, A., Bebbington, P., & Hurry, J. (1979) · Psychological Medicine · doi:10.1017/s0033291700032086 · Cited by 96
Blackwell, N., Myatt, M. A., Allafort-Duverger, T., Amour, B., Ibrahim, A., & Briend, A. (2015) · Archives of Public Health · doi:10.1186/s13690-015-0074-z · Cited by 94
“fect agreement, negative numbers indicate disagreement, with minus one being total disagreement, and zero indicating agreement expected by chance [11,12].”
BackgroundMethodSWSR: A Chinese dataset and lexicon for online sexism detection
Jiang, A., Yang, X., Liu, Y., & Zubiaga, A. (2021) · Online Social Networks and Media · doi:10.1016/j.osnem.2021.100182 · Cited by 151
Callaghan, J., Stopford, C. L., Arran, N., Boissé, M., Coleman, A., Santos, R. D., Dumas, E. M., Hart, E. P., Justo, D., Owen, G. N., et al. (2015) · Journal of Neuropsychiatry · doi:10.1176/appi.neuropsych.13070169 · Cited by 97
Sassen, S., de Booij, M., Sosef, M. N., Berendsen, R. C. M., Lammering, G., Clarijs, R., Bakker, M., Beets‐Tan, R. G. H., Warmerdam, F. A. R. M., & Vliegen, R. F. A. (2013) · European Radiology · doi:10.1007/s00330-013-2956-1 · Cited by 101
Use of the EQ-5D among patients suffering from dementia
Ankri, J., Beaufils, B., Novella, J., Morrone, I., Guillemin, F., Jolly, D., Ploton, L., & Blanchard, F. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00175-6 · Cited by 102
“Test-retest was determined by kappa coefficients [30] of the responses for categorized data after elimination ofagreement thatwould occur bychance.”
MethodFisher, T. A., & Padmawidjaja, I. A. (1999) · Journal of Multicultural Counseling and Development · doi:10.1002/j.2161-1912.1999.tb00220.x · Cited by 123
Systematic reviews to evaluate diagnostic tests
Khan, K. S., Dinnes, J., & Kleijnen, J. (2001) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/s0301-2115(00)00463-2 · Cited by 108
A Controlled Multivariate Chart Review of Multiple Complex Developmental Disorder
van der Gaag, R. J., Buitelaar, J. K., van den Ban, E. F., Bezemer, M. L., NJIO, L., & van Engeland, H. (1995) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199508000-00021 · Cited by 94
Learning how to Differ: Agreement and Reliability Statistics in Psychiatry
David, L. S. (1995) · The Canadian Journal of Psychiatry · doi:10.1177/070674379504000202 · Cited by 97
“The solution is to weight the amount of disagreement, so that a difference of many levels is penalized more than a difference of only a few levels (13).”
BackgroundComparing the Joint Agreement of Several Raters with Another Rater
Williams, G. W. (1976) · Biometrics · doi:10.2307/2529750 · Cited by 94
Genotype-free demultiplexing of pooled single-cell RNA-seq
Xu, J., Falconer, C., Nguyen, Q., Crawford, J., McKinnon, B. D., Mortlock, S., Senabouth, A., Andersen, S., Chiu, H. S., Jiang, L., et al. (2019) · Genome Biology · doi:10.1186/s13059-019-1852-7 · Cited by 94
Methodological Principles for Reproducible Performance Evaluation in Cloud Computing
Papadopoulos, A. V., Versluis, L., Bauer, A., Herbst, N., Kistowski, J. V., Ali-Eldin, A., Abad, C. L., Amaral, J. N., Tuma, P., & Iosup, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2927908 · Cited by 93
“We use the Fleiss’ Kappa analysis [66], [67] and the weighted Cohen’s Kappa analysis [68] to measure the degree of agreement between reviewers who rate a sample of a cohort; in our work, the cohort is the complete set of papers in the reviewing process.”
MethodStructural Abnormalities in Brain Magnetic Resonance Images of Depressed Children
Steingard, R. J., Renshaw, P. F., Yurgelun-Todd, D. A., APPELMANS, K. E., Lyoo, I. K., Shorrock, K. L., BUCCI, J. P., Ceseña, M., ABEBE, D., Zurakowski, D., et al. (1996) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199603000-00011 · Cited by 99
Shirk, A. J., Cushman, S. A., Waring, K. M., Wehenkel, C., Leal‐Sàenz, A., Toney, C., & López-Sánchez, C. A. (2018) · Forest Ecology and Management · doi:10.1016/j.foreco.2018.01.025 · Cited by 93
“We evaluated model performance for each method and replicate in several ways, including AUC, True Skill Statistic (TSS; Allouche et al., 2006), Cohen’s Kappa (Cohen, 1968), proportion correctly classified (PCC), specificity, and sensitivity.”
BackgroundMethodDaly, C., Neupane, B., Beyene, J., Thabane, L., Straus, S. E., & Hamid, J. S. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-024625 · Cited by 103
“The quadratic weighted kappa is equivalent to Pearson’s correlation coefficient applied to the SUCRAbased ranks, as well as Spearman’s rank correlation if it was applied to the SUCRA-based ranks or SUCRA values themselves.(32) Other weighting schemes may incorporate distances in SUCRA, or may be designed to reflect ch…”
MethodCao, J., Liu, K., Zhuo, L., Liu, L., Zhu, Y., & Peng, L. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2021.102414 · Cited by 108
Dick, P., & Jankowicz, D. (2001) · Policing An International Journal · doi:10.1108/13639510110390936 · Cited by 93
Tracheobronchoscopic assessment of exercise-induced pulmonary hemorrhage in horses
Hinchcliff, K. W., Jackson, M., Brown, J. A., Dredge, A. F., O'Callaghan, P. A., McCaffrey, J. P., Morley, P. S., Slocombe, R. F., & CLARKE, A. F. (2005) · American Journal of Veterinary Research · doi:10.2460/ajvr.2005.66.596 · Cited by 93
Greiner, B. A., Ragland, D. R., Krause, N., Syme, S. L., & Fisher, J. M. (1997) · Journal of Occupational Health Psychology · doi:10.1037//1076-8998.2.4.325 · Cited by 92
A Comparison of Systematic Screening Tools for Emotional and Behavioral Disorders
Lane, K. L., Little, M. A., Casey, A. M., Lambert, W. E., Wehby, J. H., Weisenbach, J. L., & Phillips, A. (2008) · Journal of Emotional and Behavioral Disorders · doi:10.1177/1063426608326203 · Cited by 100
Bejerholm, U., Hansson, L., & Eklund, M. (2006) · British Journal of Occupational Therapy · doi:10.1177/030802260606900203 · Cited by 91
“Statistics The interrater reliability of POES was calculated by using weighted kappa, which is the sum of the weighted frequencies corrected for chance agreement (Cohen 1968).”
MethodCOVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic
Saakyan, A., Chakrabarty, T., & Muresan, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.165 · Cited by 91
Finan, P. H., Richards, J. M., Gamaldo, C. E., Han, D., Leoutsakos, J. M., Salas, R., Irwin, M. R., & Smith, M. T. (2016) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.6262 · Cited by 90
Papini, G. B., Fonseca, P., van Gilst, M. M., Bergmans, J. W. M., Vullings, R., & Overeem, S. (2020) · Scientific Reports · doi:10.1038/s41598-020-69935-7 · Cited by 90
Manschot, S., van Passel, L., Buskens, E., Algra, A., & van Gijn, J. (1998) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.64.2.253 · Cited by 86
Influence of contextual information in emotion annotation for spoken dialogue systems
Callejas, Z., & López-Cózar, R. (2008) · Speech Communication · doi:10.1016/j.specom.2008.01.001 · Cited by 86
“To take all this information into account we have used weighted Kappa coefficients (Cohen, 1968; Fleiss and Cohen, 1973), which put the emphasis on disagreements instead of agreements.3 The computation of the weighted coefficients implies employing distance metrics between the four emotions used…”
Method
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.