Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 401–500 of 1,098
Comparison of the Null Distributions of Weighted Kappa and the C Ordinal Statistic
Cicchetti, D. V., & Fleiss, J. L. (1977) · Applied Psychological Measurement · doi:10.1177/014662167700100206 · Cited by 204
“The statistic weighted kappa was proposed by Cohen (1968) as a measure of agreement be-tween the two raters when the relative seriousness of each possible disagreement could be quantified.”
The effects of CEO equity ownership and firm diversification on risk taking
Eisenmann, T. R. (2002) · Strategic Management Journal · doi:10.1002/smj.236 · Cited by 200
Lander, M. W., Koene, B. A. S., & Linssen, S. N. (2013) · Accounting, Organizations and Society · doi:10.1016/j.aos.2012.11.001 · Cited by 193
Identifying self-admitted technical debt in open source projects using text mining
Huang, Q., Shihab, E., Xia, X., Lo, D., & Li, S. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9522-4 · Cited by 194
“They report a high level of agreement (Cohen’s Kappa coefficient (Cohen 1968) of +0.81), which makes us confident in the classification of the provided dataset.”
BackgroundInfluentialKallman, D. A., Wigley, F. M., Scott, W. W., Hochberg, M. C., & Tobin, J. D. (1989) · Arthritis & Rheumatism · doi:10.1002/anr.1780321213 · Cited by 206
Health-Related Quality of Life of Persons With Sarcoidosis
Cox, C. E., Donohue, J. F., Brown, C. D., Kataria, Y. P., & Judson, M. A. (2004) · Chest · doi:10.1378/chest.125.3.997 · Cited by 201
Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding
Dibeklioglu, H., Hammal, Z., & Cohn, J. F. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2676878 · Cited by 192
“dinal agreement above what would be expected to occur by chance [70].”
Foody, G. M., Palubinskas, G., Lucas, R. M., Curran, P. J., & Honzak, M. (1996) · Remote Sensing of Environment · doi:10.1016/s0034-4257(95)00196-4 · Cited by 198
The development and use of a standardized assessment of abnormal personality
Mann, A. H., Jenkins, R., Cutting, J. C., & Cowen, P. J. (1981) · Psychological Medicine · doi:10.1017/s0033291700041337 · Cited by 199
Cust, A. E., Smith, B. J., Chau, J., van der Ploeg, H. P., Friedenreich, C. M., Armstrong, B. K., & Bauman, A. (2008) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-5-33 · Cited by 194
Light, S. N., Coan, J. A., Zahn-Waxler, C., Frye, C., Goldsmith, H. H., & Davidson, R. J. (2009) · Child Development · doi:10.1111/j.1467-8624.2009.01326.x · Cited by 194
“Weighted kappa values (Cohen, 1968) were calculated for empathic concern, empathic cheerfulness, and empathic happiness.”
BackgroundMaternal recall of breastfeeding duration twenty years after delivery
Natland, S. T., Andersen, L. F., Nilsen, T. I. L., Forsmo, S., & Jacobsen, G. W. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-179 · Cited by 191
A deep learning interpretable classifier for diabetic retinopathy disease grading
de la Torre, J., Valls, A., & Puig, D. (2020) · Neurocomputing · doi:10.1016/j.neucom.2018.07.102 · Cited by 191
“The quadratic weighted kappa (qwk) evaluation metric [10] over the test set using a unique deep learning model without ensembling was close to the reported by human experts.”
ResultOpening Pandora's box: the unpredictability of reassurance by a normal test result
McDonald, I. G., Daly, J., Jelinek, V. M., Panetta, F., & Gutman, J. M. (1996) · BMJ · doi:10.1136/bmj.313.7053.329 · Cited by 202
Australian and U.S. News Media Portrayal of Sharks and Their Conservation
Muter, B. A., Gore, M. L., Gledhill, K. S., Lamont, C., & Huveneers, C. (2013) · Conservation Biology · doi:10.1111/j.1523-1739.2012.01952.x · Cited by 194
“We used Cohen’s kappa (Cohen 1960, 1968) to measure agreement between the 2 coders (Capozzoli et al. 1999).”
MethodResolving unfinished business: Relating process to outcome.
Greenberg, L. S., & Malcolm, W. (2002) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.70.2.406 · Cited by 184
“Interrater reliability was calculated with Cohen’s weighted kappa (Cohen, 1968), giving a coefficient of .70.”
MethodShimozuma, K., Ohashi, Y., Takeuchi, A., Aranishi, T., Morita, S., Kuroi, K., Ohsumi, S., Makino, H., Mukai, H., Katsumata, N., et al. (2009) · Supportive Care in Cancer · doi:10.1007/s00520-009-0613-7 · Cited by 193
“Hence, in this study, the concordance was examined by comparing the absolute score distribution for each grade of severity between the patient-based PNQ and the physician-based NCI-CTC using the weighted kappa coefficient [8].”
MethodThe Firm Strikes Back: Non-compete Agreements and the Mobility of Technical Professionals
Marx, M. (2011) · American Sociological Review · doi:10.1177/0003122411414822 · Cited by 198
“Cohen’s kappa (Cohen 1968) ranged from .75 to 1.0 for various indicators, with a mean of .88.”
BackgroundInfluentialHigh prevalence of metabolic syndrome in Mexico
Aguilar-Salinas, C. A., Rojas, R., Gómez-Pérez, F. J., Valles, V., Rı́os-Torres, J. M., Franco, A., Olaiz, G., Rull, J. A., & Sepúlveda, J. (2004) · Archives of Medical Research · doi:10.1016/j.arcmed.2003.06.006 · Cited by 190
Coenraads, P. J., Van Der Walle, H., Thestrup-Pedersen, K., Ruzicka, T., Dreno, B., De La Loge, C., Viala, M., Querner, S., Brown, T., & Zultak, M. (2005) · British Journal of Dermatology · doi:10.1111/j.1365-2133.2004.06270.x · Cited by 198
Kappa and Beyond: Is There Agreement?
Dettori, J. R., & Norvell, D. C. (2020) · Global Spine Journal · doi:10.1177/2192568220911648 · Cited by 171
Rau, G., & Shih, Y. (2021) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2021.101026 · Cited by 246
Role of duplex scanning for the detection of atherosclerotic renal artery disease
Hoffmann, U., Edwards, J. M., Carter, S., Goldman, M. L., Harley, J. D., Zaccardi, M. J., & Strandness, D. E. (1991) · Kidney International · doi:10.1038/ki.1991.156 · Cited by 205
Humphrey, L. L. (1989) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.57.2.206 · Cited by 193
Ando, M., Ando, Y., Hasegawa, Y., Shimokata, K., Minami, H., Wakai, K., Ohno, Y., & Sakai, S. (2001) · British Journal of Cancer · doi:10.1054/bjoc.2001.2162 · Cited by 192
“The inter-observer agreement in the assessment of PS was evaluated using simple and weighted Kappa statistics (Cohen, 1960, 1968).”
MethodA Review of Meta-Analyses in Education: Methodological Strengths and Weaknesses
Ahn, S., Ames, A. J., & Myers, N. D. (2012) · Review of Educational Research · doi:10.3102/0034654312458162 · Cited by 179
“For instance, Cooper (2009) proposed that meta-analysts should address issues that might weaken the extent to which the findings of research syntheses can be generalizable (Valentine, Cooper, Patall, Tyson, & Robinson, 2010).”
BackgroundMethodPretend Play and the Development of Emotion Regulation in Preschool Children
Galyer, K. T., & Evans, I. M. (2001) · Early Child Development and Care · doi:10.1080/0300443011660108 · Cited by 177
“92, which is considered acceptable (Cohen, 1968).”
Optimum duration of regimens for Helicobacter pylori eradication
Yuan, Y., Ford, A. C., Khan, K. J., Gisbert, J. P., Forman, D., Leontiadis, G. I., Tse, F., Calvet, X., Fallone, C., Fischbach, L., et al. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008337.pub2 · Cited by 171
Why do people ruminate in dysphoric moods?
Watkins, E., & Baracaia, S. (2001) · Personality and Individual Differences · doi:10.1016/s0191-8869(00)00053-2 · Cited by 177
“8 for all questions (Cohen, 1968).”
A critical review of emerging technologies for tackling COVID ‐19 pandemic
Mbunge, E., Akinnuwesi, B., Fashoto, S. G., Metfula, A. S., & Mashwama, P. (2021) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.237 · Cited by 184
“The degree of accuracy and reliability of quality assessment of articles was measured using Cohen Kappa statistic (Cohen, 1968), therefore, the substantial agreement of authors was 77.3%, with Cohen's k : 0.50022.”
MethodInfluentialBoyle, M. H., Furlong, W., Feeny, D., Torrance, G. W., & Hatcher, J. (1995) · Quality of Life Research · doi:10.1007/bf02260864 · Cited by 194
members, T. F. A. P., Kuhn-Thiel, A. M., Weiß, C., & Wehling, M. (2014) · Drugs & Aging · doi:10.1007/s40266-013-0146-0 · Cited by 184
“Although at first glance seemingly arbitrary, those weighing factors appear, for our purposes, plausible [28–30].”
BackgroundBoelaert, M., El-Safi, S., Hailu, A., Mukhtar, M., Rijal, S., Sundar, S., Wasunna, M., Aseffa, A., Mbui, J., Menten, J., et al. (2008) · Transactions of the Royal Society of Tropical Medicine and Hygiene · doi:10.1016/j.trstmh.2007.09.003 · Cited by 192
Trihia, H., Murray, S., Price, K., Gelber, R. D., Golouh, R., Goldhirsch, A., Coates, A. S., Collins, J., Castiglione‐Gertsch, M., & Gusterson, B. A. (2003) · Cancer · doi:10.1002/cncr.11188 · Cited by 189
“001) Positive 1 (2) 19 (40) 27 (57) — Thymidylate synthase Low 8 (12) 34 (52) 23 (35) — High 20 (11) 99 (56) 58 (33) 0.”
Stockton, D., Kellett, S., Berrios, R., Sirois, F., Wilkinson, N., & Miles, G. (2019) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465818000553 · Cited by 193
“Results were analysed using Cohen’s kappa (Cohen, 1968), before resolving any disagreements.”
MethodMehta, S., Bastero‐Caballero, R. F., Sun, Y., Zhu, R., Murphy, D. K., Hardas, B., & Koch, G. (2018) · Statistics in Medicine · doi:10.1002/sim.7679 · Cited by 193
Ready, AIM, Acquire: Impression Offsetting and Acquisitions
Graffin, S. D., Haleblian, J. (., & Kiley, J. T. (2016) · Academy of Management Journal · doi:10.5465/amj.2013.0288 · Cited by 184
“To assess inter-rater reliability we employed Cohen’s k (Cohen, 1968).”
MethodEstimating Cohesion in Small Groups Using Audio-Visual Nonverbal Behavior
Hung, H., & Gatica-Perez, D. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2055233 · Cited by 181
Eckes, T. (2005) · Language Assessment Quarterly · doi:10.1207/s15434311laq0203_2 · Cited by 188
“The weighted kappa index (Cohen, 1968) indicated that the agreement between TDN assignments on the basis of fair and observed scores was fairly high overall (.”
BackgroundSelf-rated health among U.S. adolescents
Boardman, J. D. (2006) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2005.01.006 · Cited by 184
Epidemiology of proximal humeral fractures: a detailed survey of 711 patients in a metropolitan area
Passaretti, D., Candela, V., Sessa, P., & Gumina, S. (2017) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2017.05.029 · Cited by 184
Feifel, H., & Branscomb, A. B. (1973) · Journal of Abnormal Psychology · doi:10.1037/h0034519 · Cited by 186
Schizoid personality in childhood: a controlled follow-up study
Wolff, S., & Chick, J. (1980) · Psychological Medicine · doi:10.1017/s0033291700039623 · Cited by 195
Is the World Flat or Spiky? Information Intensity, Skills, and Global Service Disaggregation
Mithas, S., & Whitaker, J. (2007) · Information Systems Research · doi:10.1287/isre.1070.0131 · Cited by 192
Bender, A. M., Lawson, D., Werthner, P., & Samuels, C. H. (2018) · Sports Medicine - Open · doi:10.1186/s40798-018-0140-5 · Cited by 165
The effectiveness of gamification in programming education: Evidence from a meta-analysis
Zhan, Z., He, L., Tong, Y., Liang, X., Guo, S., & Lan, X. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100096 · Cited by 191
Measures of interobserver agreement: Calculation formulas and distribution effects
House, A. E., House, B. J., & Campbell, M. B. (1981) · Journal of Behavioral Assessment · doi:10.1007/bf01321350 · Cited by 180
Vliegen, R. F. A., Beets, G. L., Lammering, G., Dresen, R. C., Rutten, H. J., Kessels, A. G., Oei, T.-K., de Bruïne, A. P., van Engelshoven, J. M. A., & Beets-Tan, R. G. H. (2008) · Radiology · doi:10.1148/radiol.2462070042 · Cited by 191
Exercises for mechanical neck disorders
Kay, T. M., Gross, A., Goldsmith, C. H., Hoving, J. L., & Brønfort, G. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004250.pub3 · Cited by 188
An Evidence-Based Review of Prognostic Factors for Glaucomatous Visual Field Progression
Ernest, P. J., Schouten, J. S., Beckers, H. J., Hendrikse, F., Prins, M. H., & Webers, C. A. (2013) · Ophthalmology · doi:10.1016/j.ophtha.2012.09.005 · Cited by 186
Bronchiolitis Obliterans Syndrome in Heart-Lung Transplant Recipients: Diagnosis with Expiratory CT
Bankier, A. A., Van Muylem, A., Knoop, C., Estenne, M., & Gevenois, P. A. (2001) · Radiology · doi:10.1148/radiology.218.2.r01fe09533 · Cited by 189
Cicchetti, D. V. (1981) · Applied Psychological Measurement · doi:10.1177/014662168100500114 · Cited by 186
McFadyen, A. K., Maclaren, W. M., & Webster, V. S. (2007) · Journal of Interprofessional Care · doi:10.1080/13561820701352531 · Cited by 181
“Analyses resulted in Weighted Kappa values in the range 0.102 to 0.478, with 5 items [Items 4, 5, 8, 9 and 12] not achieving an acceptable level of significance.”
MethodInfluentialPlatt, S., Weyman, A., Hirsch, S., & Hewett, S. (1980) · Social Psychiatry · doi:10.1007/bf00577960 · Cited by 183
RECOGNIZING STRONG AND WEAK OPINION CLAUSES
Wilson, T., Wiebe, J., & Hwa, R. (2006) · Computational Intelligence · doi:10.1111/j.1467-8640.2006.00275.x · Cited by 185
“There is an adaptation of Cohen’s kappa called weighted kappa (Cohen 1968), which is for use with ordinal data.”
MethodCognitive Biases in Software Engineering: A Systematic Mapping Study
Mohanani, R., Salman, I., Turhan, B., Rodriguez, P., & Ralph, P. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877759 · Cited by 158
“77) [56].”
Assessment of 2 × 2 Associations: Generalization of Signal-Detection Methodology
Kraemer, H. C. (1988) · The American Statistician · doi:10.1080/00031305.1988.10475520 · Cited by 183
“It is the weighted kappa coefficient (Cohen 1968; Fleiss 1981), where the weights reflect the context-specific costs of the two discrepancies.”
BackgroundGaziano, T. A., Abrahams-Gessel, S., Denman, C. A., Montano, C. M., Khanam, M., Puoane, T., & Levitt, N. S. (2015) · The Lancet Global Health · doi:10.1016/s2214-109x(15)00143-6 · Cited by 184
Fundamentals of clinical research for radiologists
Joseph, L., & Reinhold, C. (2005) · American Journal of Roentgenology · Cited by 193
Veigas, P. V., Callum, J., Rizoli, S., Nascimento, B., & da Luz, L. T. (2016) · Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine · doi:10.1186/s13049-016-0308-2 · Cited by 180
“The authors used EXTEM MCF ≤ 18 mm, clot lysis index at 30 min (CLI30) 71 % and APTEM MCF 7 % increase to define hyperfibrinolysis (HF) (sensitivity 100, 75, 80 % and specificity 100 % for all, respectively) in these patients with an abnormal ELT test.”
MethodInfluentialA weighted kappa statistic for reliability testing in performance analysis of sport
Robinson, G., & O’Donoghue, P. (2007) · International Journal of Performance Analysis in Sport · doi:10.1080/24748668.2007.11868383 · Cited by 175
Carlos, J. P., Wolfe, M. D., & Kingman, A. (1986) · Journal of Clinical Periodontology · doi:10.1111/j.1600-051x.1986.tb01497.x · Cited by 188
Chen, G., Faris, P., Hemmelgarn, B., Walker, R. L., & Quan, H. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-5 · Cited by 183
Cheng, C., Cheung, M. W.-L., & Wang, H.-Y. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.06.033 · Cited by 162
“Cohen (1968) kappa was computed to evaluate the level of inter-reviewer agreement for the final coding (κ=0.77).”
MethodInfluentialBrennan, P. F., & Hays, B. J. (1992) · Research in Nursing & Health · doi:10.1002/nur.4770150210 · Cited by 180
OSCI: an organisational and safety climate inventory
Silva, S., Lima, M. L., & Baptista, C. (2004) · Safety Science · doi:10.1016/s0925-7535(03)00043-2 · Cited by 168
On the pragmatic design of literature studies in software engineering: an experience-based guideline
Kuhrmann, M., Fernández, D. M., & Daneva, M. (2017) · Empirical Software Engineering · doi:10.1007/s10664-016-9492-y · Cited by 161
Disordered Eating and Unhealthy Weight Reduction Practices among Adolescent Females
Grigg, M., Bowman, J., & Redman, S. (1996) · Preventive Medicine · doi:10.1006/pmed.1996.0115 · Cited by 177
Interrater Reliability of the History and Physical Examination in Patients With Mechanical Neck Pain
Cleland, J. A., Childs, J. D., Fritz, J. M., & Whitman, J. M. (2006) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2006.06.011 · Cited by 181
Cyberbullying detection for low-resource languages and dialects: Review of the state of the art
Mahmud, T., Ptaszynski, M., Eronen, J., & Masui, F. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103454 · Cited by 158
Material Parenting: How the Use of Goods in Parenting Fosters Materialism in the Next Generation
Richins, M. L., & Chaplin, L. N. (2015) · Journal of Consumer Research · doi:10.1086/680087 · Cited by 170
“First, weighted kappa (Cohen 1968) was calculated and found to be significant for all variables.”
MethodDeep Learning Regression for Prostate Cancer Detection and Grading in Bi-Parametric MRI
Vente, C. D., Vos, P., Hosseinzadeh, M., Pluim, J., & Veta, M. (2021) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2020.2993528 · Cited by 177
“1) Voxel-Wise Predictions: We used the quadratic-weighted kappa score κ w as a metric for evaluation [41].”
MethodDevelopment of the Review Quality Instrument (RQI) for Assessing Peer Reviews of Manuscripts
van Rooyen, S., Black, N., & Godlee, F. (1999) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00047-5 · Cited by 183
“Internal consistency was assessed by means of Cronbach’s alpha [4], and test–retest reliability was assessed by means of the weighted Kappa statistic [5], the Wilcoxon signed-rank test [7], and the Friedman test [8].”
BackgroundMethodA New Interpretation of the Weighted Kappa Coefficients
Vanbelle, S. (2016) · Psychometrika · doi:10.1007/s11336-014-9439-4 · Cited by 154
“Since weighted kappa coefficients are invariant over any positive multiplicative transformation of the weights (Cohen, 1968), the unscaled form of the weights v(s)jk = | j − k|s (s ∈ N) will be used for convenience.”
BackgroundMethodInfluentialRussell, M. A., Hill, K. D., Blackberry, I., Day, L. M., & Dharmage, S. C. (2008) · Age and Ageing · doi:10.1093/ageing/afn129 · Cited by 170
“The reliability of the individual items in the FROPCom was assessed using percentage agreement and weighted Kappa [18, 19].”
MethodA generalized concordance correlation coefficient for continuous and categorical data
King, T. S., & Chinchilli, V. M. (2001) · Statistics in Medicine · doi:10.1002/sim.845 · Cited by 169
Hughes, M. A., & Garrett, D. E. (1990) · Journal of Marketing Research · doi:10.1177/002224379002700206 · Cited by 175
Why Do People Use Artificial Intelligence (AI)-Enabled Voice Assistants?
Malodia, S., Islam, N., Kaur, P., & Dhir, A. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3117884 · Cited by 201
Telemedical Retinopathy of Prematurity Diagnosis: Accuracy, Reliability, and Image Quality
Chiang, M. F. (2007) · Archives of Ophthalmology · doi:10.1001/archopht.125.11.1531 · Cited by 176
“00=near perfect agreement.(24,25) Image quality was assessed from number of “unknown” diagnoses and from image acceptability ratings by graders.”
BackgroundCeresoli, M., Coccolini, F., Catena, F., Montori, G., Di Saverio, S., Sartelli, M., & Ansaloni, L. (2016) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2015.06.012 · Cited by 168
Biederman, J., Monuteaux, M. C., Mick, E., Wilens, T. E., Fontanella, J. A., Poetzl, K. M., Kirk, T., Masse, J., & Faraone, S. V. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2005.07.009 · Cited by 172
The reproducibility and predictive value on outcome of renal biopsies from expanded criteria donors
Antonieta Azancot, M., Moreso, F., Salcedo, M., Cantarell, C., Perello, M., Torres, I. B., Montero, A., Trilla, E., Sellarés, J., Morote, J., et al. (2014) · Kidney International · doi:10.1038/ki.2013.461 · Cited by 167
Charisma perception from text and speech
Rosenberg, A., & Hirschberg, J. (2009) · Speech Communication · doi:10.1016/j.specom.2008.11.001 · Cited by 161
“We again used the weighted kappa statistic (Cohen, 1968) with quadratic weighting to determine the inter-subject agreement with respect to their assessments of each statement on each transcript.”
MethodInfluentialLand use optimization in Ningbo City with a coupled GA and PLUS model
Li, X., Fu, J., Jiang, D., Lin, G., & Cao, C. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.134004 · Cited by 165
Massage stimulates growth in preterm infants: A replication
Scafidi, F. A., Field, T. M., Schanberg, S. M., Bauer, C. R., Tucci, K., Roberts, J., Morrow, C., & Kuhn, C. M. (1990) · Infant Behavior and Development · doi:10.1016/0163-6383(90)90029-8 · Cited by 174
Diener, M. K., Rahbari, N. N., Fischer, L., Antes, G., Büchler, M. W., & Seiler, C. M. (2008) · Annals of Surgery · doi:10.1097/sla.0b013e3181724ee7 · Cited by 176
Cambach, W., Wagenaar, R. C., Koelman, T. W., van Keimpema, T., & Kemper, H. C. G. (1999) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(99)90316-7 · Cited by 176
DIAGNO II: Further Developments in a Computer Program for Psychiatric Diagnosis
SPITZER, R. L., & ENDICOTT, J. (1969) · American Journal of Psychiatry · doi:10.1176/ajp.125.7s.12 · Cited by 175
Accuracy of Self-Reported Disability in Patients With Parkinsonism
Brown, R. G., MacCarthy, B., Jahanshahi, M., & Marsden, C. D. (1989) · Archives of Neurology · doi:10.1001/archneur.1989.00520450025014 · Cited by 175
Behavior change intervention research in community settings: how generalizable are the results?
Dzewaltowski, D. A. (2004) · Health Promotion International · doi:10.1093/heapro/dah211 · Cited by 164
“The Kappa statistic (Cohen, 1968), which adjusts for chance agreement across all coding categories, was in the excellent range at 0.87.”
BackgroundInfluentialImplementing a General Framework for Assessing Interrater Agreement in Stata
Klein, D. (2018) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x1801800408 · Cited by 158
“Fleiss called his coefficient kappa, giving the wrong impression that it is a generalized version of the coefficient proposed by Cohen (1960, 1968) before.”
Efforts to establish the reliability of the Resident Assessment Instrument
Sgadari, A., Morris, J. N., Fries, B. E., Ljunggren, G., JONsson, P. V., Dupaquier, J.-N., & Schroll, M. (1997) · Age and Ageing · doi:10.1093/ageing/26.suppl_2.27 · Cited by 171
“Reliability estimates were based on the congruence between judgements of the two assessors using the weighted K [5, 6] and the Spearman-Brown intraclass correlation coefficient [7].”
MethodMainiero, L. A. (1986) · Administrative Science Quarterly · doi:10.2307/2392967 · Cited by 169
BRUGHA, T. S., JENKINS, R., TAUB, N., MELTZER, H., & BEBBINGTON, P. E. (2001) · Psychological Medicine · doi:10.1017/s0033291701004184 · Cited by 171
“If the population value of the primary concordance estimator kappa (Cohen, 1968) is 0±4, between 150 and 200 successful pairs of interviews would be required to test the null hypothesis of a zero value for kappa at the 5% significance level with 80% power.”
BackgroundMethodOn negative results when using sentiment analysis tools for software engineering research
Jongeling, R., Sarkar, P., Datta, S., & Serebrenik, A. (2017) · Empirical Software Engineering · doi:10.1007/s10664-016-9493-x · Cited by 171
“Therefore, rather than reporting accuracy of the approaches we use the Weighted kappa (Cohen 1968) and the Adjusted Rand Index (ARI) (Hubert and Arabie 1985; Santos and Embrechts 2009).”
BackgroundMethodBono, R., Blanca, M. J., Arnau, J., & Gómez‐Benito, J. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01602 · Cited by 191
“The inter-rater reliability for selection of articles was assessed with Cohen’s kappa (Cohen, 1968).”
MethodPanza, G. A., Puhl, R. M., Taylor, B. A., Zaleski, A. L., Livingston, J., & Pescatello, L. S. (2019) · PLOS ONE · doi:10.1371/journal.pone.0217623 · Cited by 163
Rectal Cancer: MR Imaging in Local Staging—Is Gadolinium-based Contrast Material Helpful?
Vliegen, R. F. A., Beets, G. L., von Meyenfeldt, M. F., Kessels, A. G. H., Lemaire, E. E. M. T., van Engelshoven, J. M. A., & Beets-Tan, R. G. H. (2005) · Radiology · doi:10.1148/radiol.2341031403 · Cited by 174
An Independent Assessment of Two Clubfoot-Classification Systems
Flynn, J. M., Donohoe, M., & Mackenzie, W. G. (1998) · Journal of Pediatric Orthopaedics · doi:10.1097/01241398-199805000-00010 · Cited by 170
Adamson, L. B., Bakeman, R., Suma, K., & Robins, D. L. (2019) · Child Development · doi:10.1111/cdev.12973 · Cited by 170
“Agreement was assessed with weighted kappas (Cohen, 1968).”
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.