Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 601–700 of 1,098
Adamson, L. B., Bakeman, R., Deckner, D. F., & Nelson, P. B. (2012) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-012-1520-1 · Cited by 131
“Agreement was assessed with weighted kappas (Cohen 1968).”
MethodA new informant‐based questionnaire for instrumental activities of daily living in dementia
Sikkes, S. A. M., de Lange‐de Klerk, E. S. M., Pijnenburg, Y. A. L., Gillissen, F., Romkes, R., Knol, D. L., Uitdehaag, B. M. J., & Scheltens, P. (2012) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2011.08.006 · Cited by 137
Report Quality of Generalized Linear Mixed Models in Psychology: A Systematic Review
Bono, R., Alarcón, R., & Blanca, M. J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.666182 · Cited by 161
“Inter-rater reliability for selection of articles was assessed with Cohen’s kappa (Cohen, 1968).”
MethodA Comparison of Reliability Coefficients for Ordinal Rating Scales
de Raadt, A., Warrens, M. J., Bosker, R. J., & Kiers, H. A. L. (2021) · Journal of Classification · doi:10.1007/s00357-021-09386-5 · Cited by 112
“…1993; Maclure and Willett 1987; Schouten 1986), while the weighted kappa coefficient (Cohen 1968) is widely used for quantifying agreement between ratings on an ordinal scale (Moradzadeh et al. 2017; Vanbelle 2016; Warrens 2012a, 2013, 2014; Vanbelle and Albert 2009; Crewson 2005; Cohen 1968).”
BackgroundMethodInfluentialSimplifying effort estimation based on Use Case Points
Ochodek, M., Nawrocki, J. R., & Kwarciak, K. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.10.005 · Cited by 145
“The inter-rater agreement in assignment of use cases to complexity classes, measured as weighted Cohen’s kappa coefficient (j) [30,31], could be interpreted as ‘‘almost perfect’’ [31] (j ranged from 0.”
Bonniaud, V., Bryant, D., Parratte, B., & Guyatt, G. H. (2008) · The Journal of Urology · doi:10.1016/j.juro.2008.08.016 · Cited by 149
Risky Play and Children’s Well-Being, Involvement and Physical Activity
Sando, O. J., Kleppe, R., & Sandseter, E. B. H. (2021) · Child Indicators Research · doi:10.1007/s12187-021-09804-5 · Cited by 165
“Using weighted kappa (Cohen 1968), inter-rater agreement was 90% for well-being, with a kappa value of 0.50.”
InfluentialStatistical Tools for Measuring Agreement
Lin, L., Hedayat, A., & Wu, W. (2011) · Springer eBooks · doi:10.1007/978-1-4614-0562-7 · Cited by 149
Individual Differences in Theory of Mind, Aggressive Behavior, and Social Skills In Young Children
Capage, L., & Watson, A. C. (2001) · Early Education and Development · doi:10.1207/s15566935eed1204_7 · Cited by 134
“Reliability was calculated on 25% of the PIPS tests using the Kappa statistic (Cohen, 1968).”
MethodAn Exploratory Study of the Formation and Impact of Electronic Service Failures1
Tan, C.-W., Benbasat, I., & Cenfetelli, R. T. (2016) · MIS Quarterly · doi:10.25300/misq/2016/40.1.01 · Cited by 140
Storli, R., & Sandseter, E. B. H. (2019) · International Journal of Play · doi:10.1080/21594937.2019.1580338 · Cited by 144
“All sequences were coded by two researchers individually, and the inter-rater agreement for the two scales were 89% for well-being with a kappa value of 0.41, and 88% for involvement with a kappa value of 0.52 (both values regarded moderate agreement according to Cohen (1968)).”
MethodInfluentialFuller, C. J., Bladon, B. M., Driver, A. J., & Barr, A. R. S. (2004) · The Veterinary Journal · doi:10.1016/j.tvjl.2004.10.012 · Cited by 144
GARCÍA‐CORTÉS, M., LUCENA, M. I., PACHKORIA, K., BORRAZ, Y., HIDALGO, R., ANDRADE, R. J., & ON BEHALF OF THE SPANISH GROUP FOR THE STUDY OF DRUG‐INDUCED LIVER DISEASE (GRUPO DE ESTUDIO PARA LAS HEPATOPATÍAS ASOCIADAS A MEDICAMENTOS, G. (2008) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2008.03655.x · Cited by 143
Testing the Difference of Correlated Agreement Coefficients for Statistical Significance
Gwet, K. L. (2015) · Educational and Psychological Measurement · doi:10.1177/0013164415596420 · Cited by 144
“Cohen (1960) proposed the kappa coefficient for two raters, and its weighted version (Cohen, 1968).”
BackgroundBrief Report: Imitation Effects on Children with Autism
Escalona, A., Field, T. M., Nadel, J., & Lundy, B. L. (2002) · Journal of Autism and Developmental Disorders · doi:10.1023/a:1014896707002 · Cited by 143
“The Kappasaveraged .72 for one third of the observations (Cohen, 1968).”
BackgroundExploring the Determinants of Broadway Show Success
Reddy, S. K., Swaminathan, V., & Motley, C. M. (1998) · Journal of Marketing Research · doi:10.1177/002224379803500307 · Cited by 169
Suh, M. H., Zangwill, L. M., Manalastas, P. I. C., Belghith, A., Yarmohammadi, A., Medeiros, F. A., Diniz‐Filho, A., Saunders, L. J., Yousefi, S., & Weinreb, R. N. (2016) · Ophthalmology · doi:10.1016/j.ophtha.2016.07.023 · Cited by 144
“To evaluate interobserver agreement for the presence of focal LC defects, the kappa coefficient was calculated.(11,31,32) All statistical analyses were performed with JMP version 11.”
MethodNeural Network-based Detection of Self-Admitted Technical Debt: From Performance to Explainability
Ren, X., Xing, Z., Xia, X., Lo, D., Wang, X., & Grundy, J. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3324916 · Cited by 178
“They reported a high level of agreement (Cohen’s Kappa coefficient [25] of 0.81), which gives us a high confidence in the SATD classification results of the provided dataset.”
BackgroundMethodInfluentialMultiple Informant Methodology: A Critical Review and Recommendations
Wagner, S. M., Rau, C., & Lindemann, E. (2010) · Sociological Methods & Research · doi:10.1177/0049124110366231 · Cited by 130
Denburg, S. D., Carbotte, R. M., & Denburg, J. A. (1987) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/01688638708405054 · Cited by 151
RELIABILITY OF CRITERION‐REFERENCED TESTS: A DECISION‐THEORETIC FORMULATION
Swaminathan, H., Hambleton, R. K., & Algina, J. J. (1974) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1974.tb00998.x · Cited by 142
Marcovici, P. A., & Taylor, G. A. (2014) · American Journal of Roentgenology · doi:10.2214/ajr.14.12636 · Cited by 136
“Interrater agreement was assessed separately for completeness and effectiveness, using a linearly weighted kappa statistic [15, 16] and percentage of agreement.”
Mustafa, R. A., Bdair, F., Akl, E. A., Garg, A. X., Thiessen‐Philbrook, H., Salameh, H., Kisra, S., NESRALLAH, G. E., Al‐Jaishi, A. A., Patel, P., et al. (2015) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.04580415 · Cited by 155
“We used Cohen k-value to measure agreement beyond chance between reviewers (11).”
MethodA Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing
Gómez‐Rodríguez, C., & Williams, P. A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.966 · Cited by 204
Romine, J. S., Sipe, J. C., Koziol, J. A., Zyroff, J., & Beutler, E. (1999) · Proceedings of the Association of American Physicians · doi:10.1046/j.1525-1381.1999.09115.x · Cited by 142
Guidelines for the clinical and economic evaluation of health care technologies
Guyatt, G. H., Drummond, M., Feeny, D. H., Tugwell, P., Stoddart, G. L., Haynes, R. B., BENNETT, K. J., & Labelle, R. J. (1986) · Social Science & Medicine · doi:10.1016/0277-9536(86)90046-8 · Cited by 142
Martinez-Calderon, J., Flores-Cortes, M., Morales-Asencio, J. M., & Luque-Suarez, A. (2019) · The Journal of Pain · doi:10.1016/j.jpain.2019.04.009 · Cited by 141
Gorson, K. C., van Schaik, I. N., Merkies, I. S. J., Lewis, R. A., Barohn, R. J., Koski, C. L., Cornblath, D. R., Hughes, R. A., Hahn, A. F., Baumgarten, M., et al. (2010) · Journal of the Peripheral Nervous System · doi:10.1111/j.1529-8027.2010.00284.x · Cited by 154
“Inter-rater reliability among the three raters was estimated using weighted kappa with confidence intervals (Cohen, 1968).”
Meeting the Burden of Proof with Case-Study Research
Taylor, J. E., Dossick, C. S., & Garvin, M. J. (2010) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0000283 · Cited by 166
“test (Bers and Smith 1990; Cohen 1968).”
THE EFFECT OF EDUCATION ON THE AUDIT EXPECTATION GAP
Monroe, G. S., & Woodliff, D. (1993) · Accounting and Finance · doi:10.1111/j.1467-629x.1993.tb00195.x · Cited by 141
Jennings, D. F., & Young, D. M. (1990) · Entrepreneurship Theory and Practice · doi:10.1177/104225879001500106 · Cited by 141
Stoodley, R., Shaw, A., & Dales, R. (1999) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(99)70266-0 · Cited by 144
Word Power: The Impact of Negative Media Coverage on Disciplining Corporate Pollution
Jia, M., Tong, L., Viswanath, P. V., & Zhang, Z. (2015) · Journal of Business Ethics · doi:10.1007/s10551-015-2596-2 · Cited by 152
Botto, E. B., Rosario, B., & Borgia, J. L. (2017) · Journal of Prosthodontics · doi:10.1111/jopr.12630 · Cited by 143
Application of deep learning for livestock behaviour recognition: A systematic literature review
Rohan, A., Rafaq, M. S., Hasan, M. J., Asghar, F., Bashir, A. K., & Dottorini, T. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109115 · Cited by 143
de Moraes, C. L., & Reichenheim, M. E. (2002) · Cadernos de Saúde Pública · doi:10.1590/s0102-311x2002000300022 · Cited by 161
“For scores, kappas with quadratic weights are used (Reliability II) (Cohen, 1968).”
Inference About Weighted Kappa in the Non-Null Case
Fleiss, J. L., & Cicchetti, D. V. (1978) · Applied Psychological Measurement · doi:10.1177/014662167800200111 · Cited by 141
A measure of agreement for use with the Holland classification system
Iachan, R. (1984) · Journal of Vocational Behavior · doi:10.1016/0001-8791(84)90001-0 · Cited by 140
Schuster, C. (2004) · Educational and Psychological Measurement · doi:10.1177/0013164403260197 · Cited by 139
“Typically, υij = 0 is selected for cells for which raters agree and υij >0i fi ≠ j, that is, raters show disagreement. Cohen (1968) defined weighted kappa as”
BackgroundInfluentialGalanis, E., Buckner, J. C., Maurer, M. J., Sykora, R., Castillo, R., Ballman, K. V., & Erickson, B. J. (2006) · Neuro-Oncology · doi:10.1215/15228517-2005-005 · Cited by 139
“Assessment of the amount of agreement among all distinct pairs of the eight measurements was summarized with observed frequency and percent, as well as with a weighted kappa statistic (Cohen, 1968) and its 95% confidence interval (CI).”
MethodComputer-Assisted Topic Classification for Mixed-Methods Social Science Research
Hillard, D., Purpura, S., & Wilkerson, J. (2008) · Journal of Information Technology & Politics · doi:10.1080/19331680801975367 · Cited by 138
“Cohen’s Kappa statistic is a standard metric used to assess interannotator reliability between two sets of results while controlling for chance agreement (Cohen, 1968).”
BackgroundNuño, L., Gómez‐Benito, J., Carmona, V. R., & Pino, Ó. (2021) · Brain Sciences · doi:10.3390/brainsci11020147 · Cited by 157
“Inter-rater reliability was as follows: mean Cohen’s [27] k = 0.”
MethodInfluentialDelayed Onset of Maternal Affection After Childbirth
Robson, K. M., & Kumar, R. (1980) · British Journal of Psychiatry · doi:10.1192/bjp.136.4.347 · Cited by 136
The Neer classification of fractures of the proximal humerus
Kristiansen, B., Andersen, U. L., Olsen, C. A., & Varmarken, J. (1988) · Skeletal Radiology · doi:10.1007/bf00361661 · Cited by 138
“This method, compared to the more commonly used chi-square method, has the advantages that the degree of disagreement is taken into account, and allowance is made for chance agreement [2, 4].”
BackgroundMethodPast and Present State Examination: the assessment of ‘lifetime ever’ psychopathology
McGuffin, P., Katz, R., & Aldrich, J. (1986) · Psychological Medicine · doi:10.1017/s0033291700009302 · Cited by 139
“Ratings of the tape-recorded interviews were performed blindly and independently, and pairwise agreement was assessed by means of Cohen's Kappa for dichotomous items and a weighted Kappa for polychotomous items (Cohen, 1968).”
MethodMarlovits, S., Striessnig, G., Kutscha-Lissberg, F., Resinger, C., Aldrian, S. M., Vécsei, V., & Trattnig, S. (2004) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-004-0535-3 · Cited by 137
“To evaluate the validity and reliability of the scoring, the MRI scans were read by two independent radiologists, and the interobserver variability was determined using kappa statistics (kappa [j] and weighted kappa [jw]; [6, 9]).”
BackgroundMethodMcElwain, N. L., & Volling, B. L. (2005) · Journal of Family Psychology · doi:10.1037/0893-3200.19.4.486 · Cited by 137
Northern Hemisphere vegetation change drives a Holocene thermal maximum
Thompson, A. J., Zhu, J., Poulsen, C. J., Tierney, J. E., & Skinner, C. B. (2022) · Science Advances · doi:10.1126/sciadv.abj6535 · Cited by 149
Material possessions as stereotypes: Material images of different socio-economic groups
Dittmar, H. (1994) · Journal of Economic Psychology · doi:10.1016/0167-4870(94)90011-6 · Cited by 130
Shipp, A. J., Furst‐Holloway, S., Harris, T. B., & Rosen, B. (2013) · Personnel Psychology · doi:10.1111/peps.12039 · Cited by 158
Rahbari, N. N., Wente, M. N., Schemmer, P., Diener, M. K., Hoffmann, K., Motschall, E., Schmidt, J., Weitz, J., & Büchler, M. W. (2008) · British Journal of Surgery · doi:10.1002/bjs.6141 · Cited by 137
Berkel, D. V., Tabrizian, P., Dorning, M. A., Smart, L. S., Newcomb, D., Mehaffey, M. H., Neale, A., & Meentemeyer, R. K. (2018) · Ecosystem Services · doi:10.1016/j.ecoser.2018.03.022 · Cited by 137
“…we compared the land use/cover shown in the images with the land use/cover that was determined to be visible from the viewshed of each point, calculating percentage agreement and Cohen Kappa Coefficient, which accounts for the possibility of the agreement occurring by chance (Cohen, 1968).”
In-session change in emotionally focused therapy.
Greenberg, L. S., Ford, C. L., Alden, L. S., & Johnson, S. M. (1993) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.61.1.78 · Cited by 128
Baker, P., & Potts, A. (2013) · Critical Discourse Studies · doi:10.1080/17405904.2012.744320 · Cited by 135
“Inter-rater agreement was also measured using Cohen’s chance-correlated weighted kappa (kw) coefficient, which allows for assignment of varying weights to denote disagreement ‘seriousness’ in ordered categories (Cohen, 1968).”
MethodInfluentialEvaluation of a clinical test. I: Assessment of reliability
Kha, K. S., & Chien, P. F. W. (2001) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2001.00150.x · Cited by 135
BRUGHA, T. S., BEBBINGTON, P. E., JENKINS, R., MELTZER, H., TAUB, N. A., JANAS, M., & VERNON, J. (1999) · Psychological Medicine · doi:10.1017/s0033291799008892 · Cited by 135
“Power calculations for kappa (Cohen, 1968) showed that at least 150 successful pairs of interviews would provide 80% power to detect a non-zero kappa at the 5% significance level, if the true population value of kappa was 0±4, given at least a 4% ‘prevalence’ of disorder in the second phase sample.”
BackgroundMethodInfluentialAnalysis of flexible decision strategies for sustainability-focused green product recovery system
Mangla, S. K., Madaan, J., & Chan, F. T. S. (2013) · International Journal of Production Research · doi:10.1080/00207543.2013.774493 · Cited by 140
“The statistics measure, known as the weighted kappa, measures the consistency among judgments across all categories and was also given by Cohen (1968).”
BackgroundCategorizing Stroke Prognosis Using Different Stroke Scales
Govan, L., Langhorne, P., & Weir, C. J. (2009) · Stroke · doi:10.1161/strokeaha.109.557645 · Cited by 135
A new multi-objective wrapper method for feature selection – Accuracy and stability analysis for BCI
González, J., Ortega, J., Damas, M., Martín-Smith, P., & Gan, J. Q. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.017 · Cited by 142
AN EMPIRICAL INVESTIGATION OF THE AUDIT EXPECTATION GAP: AUSTRALIAN EVIDENCE
Monroe, G. S., & Woodliff, D. (1994) · Accounting and Finance · doi:10.1111/j.1467-629x.1994.tb00262.x · Cited by 135
Duncan, L. M., Berwick, M., BRUIJN, J. A., Byers, H. R., Mihm, M. C., & Barnhill, R. L. (1993) · Journal of Investigative Dermatology · doi:10.1038/jid.1993.55 · Cited by 135
“Originally described by Cohen [14], the higher the kappa value, the better the agreement.”
BackgroundLandis, J. R., & Koch, G. G. (1975) · Statistica Neerlandica · doi:10.1111/j.1467-9574.1975.tb00254.x · Cited by 133
Brodsky, J. E., Brooks, P. J., Scimeca, D., Todorova, R., Galati, P., Batson, M., Grosso, R., Matthews, M., Miller, V., & Caulfield, M. (2021) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-021-00291-4 · Cited by 93
“93 as a measure of inter-rater agreement, which takes into account the closeness of ratings (Cohen, 1968).”
BackgroundMethodInfluentialMeasures of clinical agreement for nominal and categorical data: The kappa coefficient
Cyr, L., & Francis, K. T. (1992) · Computers in Biology and Medicine · doi:10.1016/0010-4825(92)90063-s · Cited by 146
Florsheim, P., Tolan, P. H., & Gorman-Smith, D. (1996) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.64.6.1222 · Cited by 134
Stansbury, K., & Sigman, M. (2000) · The Journal of Genetic Psychology · doi:10.1080/00221320009596705 · Cited by 122
“75, acceptable based on Cohen’s standards (Cohen, 1968).”
Kelley, G. A., & Kelley, K. S. (2006) · Metabolism · doi:10.1016/j.metabol.2006.06.021 · Cited by 151
“86 using Cohen’s [9] j coefficient.”
Reliability and validity of triage systems in paediatric emergency care
van Veen, M., & Moll, H. A. (2009) · Scandinavian Journal of Trauma Resuscitation and Emergency Medicine · doi:10.1186/1757-7241-17-38 · Cited by 134
“[27] If the inter-rater agreement between multiple observers is studied, the intraclass correlation coefficient (ICC) can be used.”
Tooth, L. R., & Ottenbacher, K. J. (2004) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2003.12.002 · Cited by 134
Park, M. J., Kim, S. H., Lee, S. J., Jang, K. M., & Rhim, H. (2011) · Radiology · doi:10.1148/radiol.11102135 · Cited by 133
Wilson, T. (2008) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 179
Clinical diagnosis and prognosis following spinal cord injury
Burns, A. S., Marino, R. J., Flanders, A. E., & Flett, H. M. (2012) · Handbook of clinical neurology · doi:10.1016/b978-0-444-52137-8.00003-6 · Cited by 133
Spriggs, J. F. (1996) · American Journal of Political Science · doi:10.2307/2111745 · Cited by 133
Dhaliwal, H., Hoeroldt, B., Dube, A. K., McFarlane, E., Underwood, J. C. E., Karajeh, M. A., & Gleeson, D. C. (2015) · The American Journal of Gastroenterology · doi:10.1038/ajg.2015.139 · Cited by 142
Accuracy of High-Resolution Magnetic Resonance Imaging in Preoperative Staging of Rectal Cancer
Akasu, T., Iinuma, G., Takawa, M., Yamamoto, S., Muramatsu, Y., & Moriyama, N. (2009) · Annals of Surgical Oncology · doi:10.1245/s10434-009-0613-3 · Cited by 143
Structured Interview for Assessing Perceptual Anomalies (SIAPA)
Bunney, W. E., Hetrick, W. P., Bunney, B. G., Patterson, J. V., Jin, Y., Potkin, S. G., & Sandman, C. A. (1999) · Schizophrenia Bulletin · doi:10.1093/oxfordjournals.schbul.a033402 · Cited by 129
“Interrater agreement for individual items was calculated using the weighted kappa coefficient (Cohen 1968).”
MethodCytrynbaum, S., Ginath, Y., Birdwell, J., & Brandt, L. (1979) · Evaluation Review · doi:10.1177/0193841x7900300102 · Cited by 133
Sitting and standing intention can be decoded from scalp EEG recorded prior to movement execution
Bulea, T. C., Prasad, S., Kilicarslan, A., & Contreras-Vidal, J. L. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00376 · Cited by 140
Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices
Feng, G. C. (2014) · Methodology · doi:10.1027/1614-2241/a000086 · Cited by 144
Abd‐El‐Khalick, F., Myers, J. Y., Summers, R., Brunner, J. L., Waight, N., Wahbeh, N., Zeineddin, A. A., & Belarmino, J. J. (2016) · Journal of Research in Science Teaching · doi:10.1002/tea.21339 · Cited by 148
Developing quality fidelity and engagement measures for complex health interventions
Walton, H., Spector, A. E., Williamson, M., Tombor, I., & Michie, S. (2019) · British Journal of Health Psychology · doi:10.1111/bjhp.12394 · Cited by 146
“Reliability was calculated using Cohen’s weighted kappa and percentage agreement (Cohen, 1968; Gwet, 2014).”
MethodPatterns of death among avalanche fatalities: a 21-year review
Boyd, J. J., Haegeli, P., Abu‐Laban, R. B., Shuster, M., & Butt, J. C. (2009) · Canadian Medical Association Journal · doi:10.1503/cmaj.081327 · Cited by 132
Comorbidity measures for stroke outcome research: A preliminary study
Liu, M., Domen, K., & Chino, N. (1997) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(97)90259-8 · Cited by 133
Uterine polypectomy in the management of abnormal uterine bleeding: A systematic review
Nathani, F., & Clark, T. J. (2006) · Journal of Minimally Invasive Gynecology · doi:10.1016/j.jmig.2006.03.015 · Cited by 131
PROBA-V mission for global vegetation monitoring: standard products and image quality
Dierckx, W., Sterckx, S., Benhadj, I., Livens, S., Duhoux, G., Achteren, T. V., François, M., Mellab, K., & Saint, G. (2014) · International Journal of Remote Sensing · doi:10.1080/01431161.2014.883097 · Cited by 131
“…(amount of correct classification); ● kappa statistics (Cohen’s kappa coefficient, sometimes proposed as the standard of classification accuracy; see Cohen 1968 and Smits, Dellepiane, and Schowengerdt 1999); ● percentage of cloud over-detection (amount of false positives); ● percentage of…”
BackgroundThe Scientific Basis of Guideline Recommendations on Sugar Intake: A Systematic Review
Erickson, J., Sadeghirad, B., Lytvyn, L., Slavin, J., & Johnston, B. C. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2020 · Cited by 127
Tangible symbols, tangible outcomes
Rowland, C., & Schweigert, P. D. (2000) · Augmentative and Alternative Communication · doi:10.1080/07434610012331278914 · Cited by 131
“A score greater than .60 is preferred (Cohen, 1960, 1969).”
BackgroundPersonius, K. E., Pandya, S., King, W. M., Tawil, R., McDermott, M., & The FSH DY Group (1994) · Physical Therapy · doi:10.1093/ptj/74.3.253 · Cited by 131
Kelley, G. A., Kelley, K. S., Hootman, J. M., & Jones, D. L. (2011) · Arthritis Care & Research · doi:10.1002/acr.20347 · Cited by 121
A Measure of Agreement for Interval or Nominal Multivariate Observations
Janson, H., & Olsson, U. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971239 · Cited by 130
Interobserver and intraobserver variation in the assessment of antepartum cardiotocograms
Lotgering, F. K., Wallenburg, H. C. S., & Schouten, H. J. A. (1982) · American Journal of Obstetrics and Gynecology · doi:10.1016/0002-9378(82)90440-9 · Cited by 130
Parenting behaviors of women who were sexually abused as children in their families of origin
BURKETT, L. P. (1991) · Family Process · doi:10.1111/j.1545-5300.1991.00421.x · Cited by 129
Usher, M., & Barak, M. (2018) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2017.1405238 · Cited by 121
Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. Β. (2020) · Journal of the Learning Sciences · doi:10.1080/10508406.2020.1783269 · Cited by 143
Role of Ultrasound in the Assessment of Vocal Cord Function in Infants and Children
Friedman, E. M. (1997) · Annals of Otology Rhinology & Laryngology · doi:10.1177/000348949710600304 · Cited by 129
Coherent Citation-Based Summarization of Scientific Papers
Abu-Jbara, A., & Radev, D. (2011) · Cited by 176
CASIE: Extracting Cybersecurity Event Information from Text
Satyapanich, T., Ferraro, F., & Finin, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6401 · Cited by 112
“We measured the inter-annotator agreement using Cohen’s Kappa score (Cohen 1968).”
MethodAlva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., & Specia, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.424 · Cited by 91
Evaluation of a SARS-CoV-2 rapid antigen test: Potential to help reduce community spread?
Toptan, T., Eckermann, L., Pfeiffer, A., Hoehl, S., Ciesek, S., Drosten, C., & Corman, V. M. (2020) · Journal of Clinical Virology · doi:10.1016/j.jcv.2020.104713 · Cited by 128
Yang, F. X., Li, X., Lau, V. M.-C., & Zhu, V. Z. (2021) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2020.102855 · Cited by 137
“To assess the inter-rater reliability between different coders, Cohen’s Kappa was calculated (Cohen, 1968).”
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.