Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 501–600 of 1,098
Pauley, D., Cuijpers, P., Papola, D., Miguel, C., & Karyotaki, E. (2021) · Psychological Medicine · doi:10.1017/s0033291721001999 · Cited by 170
“Interrater reliability was calculated and reported using the interclass correlation coefficient (ICC) and Cohen ’ s Kappa (Banerjee, Capozzoli, McSweeney, & Sinha, 1999; Barnhart, Haber, & Lin, 2007; Cohen, 1968).”
Motivation gains of inferior group members: A meta-analytical review.
Weber, B., & Hertel, G. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.93.6.973 · Cited by 158
“The overall mean difference between working under conjunctive group conditions and working individually is g .72 (d .82; k 43; CI .63, .81; Qw 122.52, p .01), which can be considered a large effect ( Cohen, 1968 ).”
BackgroundInfluentialHoster, E., Klapper, W., Hermine, O., Kluin-Nelemans, H. C., Walewski, J., van Hoof, A., Trneny, M., Geisler, C. H., Di Raimondo, F., Szymczyk, M., et al. (2014) · Journal of Clinical Oncology · doi:10.1200/jco.2013.52.2466 · Cited by 166
von Allmen, D., Davidoff, A. M., London, W. B., Van Ryn, C., Haas-Kogan, D. A., Kreissman, S. G., Khanna, G., Rosen, N., Park, J. R., & La Quaglia, M. P. (2017) · Journal of Clinical Oncology · doi:10.1200/jco.2016.67.2642 · Cited by 167
Ueno, Y., Forghani, B., Forghani, R., Dohan, A., Zeng, X. Z., Chamming’s, F., Arseneau, J., Fu, L., Gilbert, L., Gallix, B., et al. (2017) · Radiology · doi:10.1148/radiol.2017161950 · Cited by 166
Silva-Rodríguez, J., Colomer, A., Sales, M. A., Molina, R., & Naranjo, V. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105637 · Cited by 165
“In the literature, this fact is addressed using the Cohen’s quadratic kappa ( κ ) metric [33].”
MethodVisual health assessment of North Atlantic right whales (Eubalaena glacialis) using photographs
Pettis, H. M., Rolland, R. M., Hamilton, P. K., Brault, S., Knowlton, A. R., & Kraus, S. D. (2004) · Canadian Journal of Zoology · doi:10.1139/z03-207 · Cited by 162
“Pa-rameter entries that were scored as non-X (i.e., photographs were of sufficient quality to score the parameter) by the pri - agreement (Cohen 1968). independently for the right and left sides of the heads.”
MethodNeurobiology of culturally common maternal responses to infant cry
Bornstein, M. H., Putnick, D. L., Rigo, P., Esposito, G., Swain, J. E., Suwalsky, J. T. D., Su, X., Du, X., Zhang, K., Cote, L. R., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1712022114 · Cited by 168
Aaltonen, J., Seikkula, J., & Lehtinen, K. (2011) · Psychosis · doi:10.1080/17522439.2011.601750 · Cited by 161
“We used the kappa coefficient as a measure of diagnostic reliability (Cohen, 1968).”
MethodLahaye, M. J., Engelen, S. M. E., Kessels, A. G. H., de Bruïne, A. P., von Meyenfeldt, M. F., van Engelshoven, J. M. A., van de Velde, C. J. H., Beets, G. L., & Beets-Tan, R. G. H. (2008) · Radiology · doi:10.1148/radiol.2463070221 · Cited by 169
BLEU is Not Suitable for the Evaluation of Text Simplification
Sulem, E., Abend, O., & Rappoport, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1081 · Cited by 144
“We then compute weighted quadratic κ (Cohen, 1968) between Annotator 1 and 2.”
MethodMeme marketing: How can marketers drive better engagement using viral memes?
Malodia, S., Dhir, A., Bilgihan, A., Sinha, P., & Tikoo, T. (2022) · Psychology and Marketing · doi:10.1002/mar.21702 · Cited by 197
Marchese Robinson, R. L., Palczewska, A., Palczewski, J., & Kidley, N. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00753 · Cited by 167
Statistical methods in epidemiology. v. Towards an understanding of the kappa coefficient
Rigby, A. S. (2000) · Disability and Rehabilitation · doi:10.1080/096382800296575 · Cited by 161
Disturbed voluntary motor activity in schizophrenic disorder
Manschreck, T. C., Maher, B. A., Rucklos, M. E., & Vereen, D. R. (1982) · Psychological Medicine · doi:10.1017/s0033291700043300 · Cited by 166
Wright, P. M., & Craig, M. W. (2011) · Measurement in Physical Education and Exercise Science · doi:10.1080/1091367x.2011.590084 · Cited by 195
“This statistic addresses the critique that percent agreement alone is not a sufficient measure for inter-rater reliability, as it does not factor chance into the calculations (Cohen, 1968); however, application of the Kappa statistic proved inappropriate in this study for two reasons.”
MethodNallamothu, B. K., Spertus, J. A., Lansky, A. J., Cohen, D. J., Jones, P. G., Kureshi, F., Dehmer, G. J., Drozda, J. P., Walsh, M. N., Brush, J. E., et al. (2013) · Circulation · doi:10.1161/circulationaha.113.001952 · Cited by 159
Watson, P. L., Pandharipande, P., Gehlbach, B. K., Thompson, J. L., Shintani, A. K., Dittus, B. S., Bernard, G. R., Malow, B. A., & Ely, E. W. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e31828a3f75 · Cited by 166
“To assess interrater reliability of the proposed sleep scoring classifications, we used the weighted kappa statistic to account for nearness of agreement (34).”
MethodFears and phobias in women: A community study.
Costello, C. G. (1982) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.91.4.280 · Cited by 164
Dresen, R. C., Beets, G. L., Rutten, H. J. T., Engelen, S. M. E., Lahaye, M. J., Vliegen, R. F. A., de Bruïne, A. P., Kessels, A. G. H., Lammering, G., & Beets-Tan, R. G. H. (2009) · Radiology · doi:10.1148/radiol.2521081200 · Cited by 167
Comparison of self-report and structured clinical interview in the identification of depression
Stuart, A. L., Pasco, J. A., Jacka, F. N., Brennan-Olsen, S. L., Berk, M., & Williams, L. J. (2014) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2013.12.019 · Cited by 182
“Calculations for the self-report data relative to the SCID-I/NP data included sensitivity — calculated by dividing the number of true positives by the number of true positives and false negatives; specificity — calculated by dividing the number of true negatives by the number of true negatives and false positives; pos…”
MethodSemEval 2016 Task 11: Complex Word Identification
Paetzold, G., & Specia, L. (2016) · Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) · doi:10.18653/v1/s16-1085 · Cited by 158
“The Cohen’s Kappa coefficient (Cohen, 1968) was not used due to the large disparity between the number of complex and simple words, which causes the likelihood of annotators agreeing by chance to be higher than the relative observed agreement.”
MethodAbraham, R. E., Van, H. L., Foeken, I. V., Ingenhoven, T. J. M., Tremonti, W., Vries, I. P.-D., Groot, J. D., Spijker, A. V. ‘., & Spinhoven, P. (2001) · Journal of Personality Disorders · doi:10.1521/pedi.15.5.457.19198 · Cited by 166
Storch, E. A., Lewin, A. B., De Nadai, A. S., & Murphy, T. K. (2010) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2010.04.005 · Cited by 158
Zimmermann, J., Ehrenthal, J. C., Cierpka, M., Schauenburg, H., Doering, S., & Benecke, C. (2012) · Journal of Personality Assessment · doi:10.1080/00223891.2012.700664 · Cited by 162
“The authors used weighted kappa (Cohen, 1968) to assess the correspondence between two raters.”
MethodEvaluating Agreement: Conducting a Reliability Study
on Behalf of the Collaboration for Outcome Assessment in Surgical Trials (COAST) Musculoskeletal Group*, Karanicolas, P. J., Bhandari, M., Kreder, H., Moroni, A., Richardson, M., Walter, S. D., Norman, G. R., & Guyatt, G. H. (2009) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.h.01624 · Cited by 160
Neale, J., Vitoratou, S., Finch, E., Lennon, P., Mitcheson, L., Panebianco, D., Rose, D., Strang, J., Wykes, T., & Marsden, J. (2016) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2016.06.006 · Cited by 150
Modeling Agreement among Raters
Tanner, M. A., & Young, M. A. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477157 · Cited by 161
Benzodiazepines in the treatment of catatonic syndrome
Ungvari, G. S., Leung, C. M., Wong, M. K., & Lau, J. (1994) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1994.tb01515.x · Cited by 166
Didactic methods for enhancing children's peer relations: A quantitative review
Schneider, B. H. (1992) · Clinical Psychology Review · doi:10.1016/0272-7358(92)90142-u · Cited by 162
Clare, A. W., & Cairns, V. E. (1978) · Psychological Medicine · doi:10.1017/s0033291700018808 · Cited by 163
Precision of Health-Related Quality-of-Life Data Compared With Other Clinical Measures
Hahn, E. A., Cella, D., Chassany, O., Fairclough, D. L., Wong, G. Y., & Hays, R. D. (2007) · Mayo Clinic Proceedings · doi:10.4065/82.10.1244 · Cited by 160
The Prevalence and 3-Year Incidence of Dementia in Older Black and White Community Residents
Fillenbaum, G. G., Heyman, A., Huber, M. S., Woodbury, M. A., Leiss, J., Schmader, K. E., Bohannon, A., & Trapp-Moen, B. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00024-9 · Cited by 164
Juvenile rheumatoid arthritis: Benefits from massage theraphy
Field, T., Hernandez-Reif, M., Seligmen, S., Krasnegor, J., Sunshine, W., Rivas-Chacon, R., Schanberg, S., & Kuhn, C. (1997) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/22.5.607 · Cited by 162
“Interrater reliability was calculated using Cohen's kappa, a chance-corrected statistic (Cohen, 1968) on 20% of the observations.”
MethodContact in adoption and adoptive identity formation: The mediating role of family conversation.
Korff, L. V., & Grotevant, H. D. (2011) · Journal of Family Psychology · doi:10.1037/a0023388 · Cited by 179
An exploration of symmetry in early autism spectrum disorders: Analysis of lying
Esposito, G., Venuti, P., Maestro, S., & Muratori, F. (2008) · Brain and Development · doi:10.1016/j.braindev.2008.04.005 · Cited by 172
Towards Exploiting Background Knowledge for Building Conversation Systems
Moghe, N., Arora, S., Banerjee, S., & Khapra, M. M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1255 · Cited by 157
“We computed inter-annotator agreement using the mean linearly weighted Cohen’s κ (Cohen, 1968) and mean Krippendorff’s α (Hayes and Krippendorff, 2007).”
MethodSex matters: Modeling male and female habitat differences for jaguar conservation
Conde, D. A., Colchero, F., Zarza, H., Christensen, N. L., Sexton, J. O., Manterola, C., Chávez, C., Rivera, A., Azuara, D., & Ceballos, G. (2010) · Biological Conservation · doi:10.1016/j.biocon.2010.04.049 · Cited by 159
“…Sex matters: Modeling m (2010), doi:10.1016/j.biocon.2010.04.049 a measure of the accuracy improvement of the model over that expected by chance (Cohen, 1968; Guisan and Zimmermann, 2000), and is appropriate when the proportions of positive and negative cases is similar (Fielding and Bell,…”
BackgroundMethodInfluentialEstimating the extent and structure of trade in horticultural orchids via social media
Hinsley, A., Lee, T. E., Harrison, J. R., & Roberts, D. L. (2016) · Conservation Biology · doi:10.1111/cobi.12721 · Cited by 161
Polite Dialogue Generation Without Parallel Data
Niu, T., & Bansal, M. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00027 · Cited by 158
Reliability of reviews for the American Psychologist: A biostatistical assessment of the data.
Cicchetti, D. V. (1980) · American Psychologist · doi:10.1037//0003-066x.35.3.300 · Cited by 160
Shin, D. W., Cho, J., Roter, D. L., Kim, S. Y., Sohn, S. K., Yoon, M., Kim, Y., Cho, B., & Park, J. (2013) · Psycho-Oncology · doi:10.1002/pon.3339 · Cited by 142
Sample Size Determinations for the Two Rater Kappa Statistic
Flack, V. F., Afifi, A. A., Lachenbruch, P. A., & Schouten, H. J. A. (1988) · Psychometrika · doi:10.1007/bf02294215 · Cited by 153
What Can Virtual Reality Teach Us About Prosocial Tendencies in Real and Virtual Environments?
Gillath, O., McCall, C., Shaver, P. R., & Blascovich, J. (2008) · Media Psychology · doi:10.1080/15213260801906489 · Cited by 149
Maternal Chronological Age, Prenatal and Perinatal History, Social Support, and Parenting of Infants
Bornstein, M. H., Putnick, D. L., Suwalsky, J. T. D., & Gini, M. (2006) · Child Development · doi:10.1111/j.1467-8624.2006.00908.x · Cited by 150
“For all behaviors that were continuously coded, kappa (κ; Cohen, 1960, 1968) was used as the index of coder reliability; for time-sampled behaviors, the Intra-Class Correlation (ICC: McGraw & Wong, 1996) was used.”
MethodJolliffe, L. J., Lannin, N. A., Cadilhac, D. A.-M., & Hoffmann, T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-018791 · Cited by 178
Racial differences in the diagnosis of psychosis
Strakowski, S. M., Flaum, M., Amador, X., Bracha, H. S., Pandurangi, A. K., Robinson, D., & Tohen, M. (1996) · Schizophrenia Research · doi:10.1016/0920-9964(96)00041-2 · Cited by 158
“Thus, for this particular analysis, in which we wished to determine whether overall symptom differences were present between racial groups, and then identify which symptoms most contributed to that difference, specifically whether first-rank symptoms contributed to the difference, the use of a MANCOVA followed by the…”
MethodAl-Bouwarthan, M., Quinn, M. M., Kriebel, D., & Wegman, D. H. (2019) · Annals of Work Exposures and Health · doi:10.1093/annweh/wxz033 · Cited by 158
Sleep Staging Based on Autonomic Signals: A Multi-Center Validation Study
Hedner, J. A., White, D. P., Malhotra, A., Herscovici, S., Pittman, S. D., Zou, D., Grote, L., & Pillar, G. (2011) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.1078 · Cited by 170
Waldinger, R. J., Schulz, M. S., Hauser, S. T., Allen, J. P., & Crowell, J. A. (2004) · Journal of Family Psychology · doi:10.1037/0893-3200.18.1.58 · Cited by 138
“Interrater reliability for all SPAFF codes as calculated using Cohen’s kappa (Cohen, 1968) was .”
MethodInfluentialvan den Berg, R., Lenczner, G., Feydy, A., van der Heijde, D. M. F. M., Reijnierse, M., Saraux, A., Rahmouni, A., Dougados, M., & Claudepierre, P. (2014) · Arthritis & Rheumatology · doi:10.1002/art.38738 · Cited by 158
“Agreement was calculated using cross-tabulation expressed in Cohen’s kappa or linear weighted kappa ( W) as appropriate, agreement on the positive cases (positive agreement), and agreement on the negative cases (negative agreement) for the following comparisons (14,15): interreader agreement on the presence/absence of…”
MethodValidation of automated scoring of science assessments
Liu, O. L., Rios, J. A., Heilman, M., Gerard, L., & Linn, M. C. (2016) · Journal of Research in Science Teaching · doi:10.1002/tea.21299 · Cited by 144
The effects of social motives on behavior in social dilemmas in two cultures
Liebrand, W. B. G., & van Run, G. J. (1985) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(85)90008-3 · Cited by 158
“Using the disagreement weights, 0 (diagonal), 1 (adjacent diagonals), 4 (next diagonals), and 9 (upper right and lower left corner), the weighted K coefficient (Cohen, 1968), is .”
Pragmatic Aspects of the Base-rate Fallacy
Macchi, L. (1995) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640749508401384 · Cited by 158
Deyo, R. A., Inui, T. S., Leininger, J. D., & Overman, S. S. (1983) · Medical Care · doi:10.1097/00005650-198302000-00006 · Cited by 158
Reliability of Psychiatric Diagnosis: I. A Methodological Review
Helzer, J. E. (1977) · Archives of General Psychiatry · doi:10.1001/archpsyc.1977.01770140019001 · Cited by 158
McFadyen, A. K., Webster, V. S., & Maclaren, W. M. (2006) · Journal of Interprofessional Care · doi:10.1080/13561820600991181 · Cited by 157
“The test-retest pairs for each individual Likert scale item within each instrument were analysed using a Weighted Kappa coefficient (Cohen, 1968) or if appropriate, when the scores were only dichotomous, the simple Kappa coefficient (Cohen, 1960).”
MethodAn Experimental Investigation of the Innovation Capabilities of Undergraduate Engineering Students
Genco, N., Hölttä‐Otto, K., & Seepersad, C. C. (2012) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2012.tb00041.x · Cited by 180
Crewson, P. E. (2005) · American Journal of Roentgenology · doi:10.2214/ajr.184.5.01841391 · Cited by 156
“AJR:184, May 2005 1393 Confidence Intervals For estimating confidence intervals, a different formula is used for SE [10].”
MethodInfluentialComparing case-based reasoning classifiers for predicting high risk software components
El Emam, K., Benlarbi, S., Goel, N., & Rai, S. N. (2001) · Journal of Systems and Software · doi:10.1016/s0164-1212(00)00079-0 · Cited by 160
“Cohen also defined a weighted version of Kappa [ 15 ], but the characteristics that are of interest to us are the same for Kappa and its weighted version.”
BackgroundChildren with Autism Display more Social Behaviors after Repeated Imitation Sessions
Field, T., Field, T., Sanders, C., & Nadel, J. (2001) · Autism · doi:10.1177/1362361301005003008 · Cited by 150
Native and Nonnative Teachers of L2 Pronunciation: Effects on Learner Performance
Levis, J. M., Sonsaat, S., Link, S., & Barriuso, T. A. (2016) · TESOL Quarterly · doi:10.1002/tesq.272 · Cited by 199
Health literacy of caregivers of adult care recipients: A systematic scoping review
Yuen, E. Y. N., Knight, T., Ricciardelli, L. A., & Burney, S. (2016) · Health & Social Care in the Community · doi:10.1111/hsc.12368 · Cited by 158
Wagner, F. H., Ferreira, M. P., Sánchez, A., Hirye, M. C. M., Zortea, M., Gloor, E., Phillips, O. L., Filho, C. R. D. S., Shimabukuro, Y. E., & Aragão, L. E. O. C. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2018.09.013 · Cited by 163
“From the confusion matrix, the Kappa index and overall accuracy were computed (Cohen, 1968; Foody, 2002).”
MethodRenaud, D. L., Buss, L., Wilms, J. N., & Steele, M. A. (2020) · Journal of Dairy Science · doi:10.3168/jds.2020-18907 · Cited by 134
A rating scale for aggressive behaviour in the elderly – the RAGE
Patel, V., & Hope, R. A. (1992) · Psychological Medicine · doi:10.1017/s0033291700032876 · Cited by 157
The factors associated with the burnout syndrome and fatigue in Cypriot nurses: a census report
Raftopoulos, V., Charalambous, A., & Talias, Μ. A. (2012) · BMC Public Health · doi:10.1186/1471-2458-12-457 · Cited by 183
“The items were considered adequate if there was> 90% agreement, based on the use of kappa statistics [37].”
MethodRider, L. G., Miller, F. W., Feldman, B. M., Perez, M. D., Rennebohm, R. M., Lindsley, C. B., Zemel, L. S., Wallace, C. A., Ballinger, S. H., Bowyer, S. L., et al. (1997) · Arthritis & Rheumatism · doi:10.1002/art.1780401109 · Cited by 167
Accuracy of Gadofosveset-enhanced MRI for Nodal Staging and Restaging in Rectal Cancer
Lambregts, D. M. J., Beets, G. L., Maas, M., Kessels, A. G. H., Bakers, F. C. H., Cappendijk, V. C., Engelen, S. M. E., Lahaye, M. J., de Bruïne, A. P., Lammering, G., et al. (2011) · Annals of Surgery · doi:10.1097/sla.0b013e31820b01f1 · Cited by 156
Asthma inhaler adherence determinants in adults: systematic review of observational data
Dima, A. L., Hernández, G., Cunillera, O., Ferrer, M., de Bruin, M., & the ASTRO-LAB group (2014) · European Respiratory Journal · doi:10.1183/09031936.00172114 · Cited by 164
“Reliability was assessed by estimating inter-rate agreement with weighted kappa, considered appropriate for ordinal scores [28], and interpreted based on established thresholds for poor, fair, moderate, good and excellent agreement (0.20, 0.40, 0.60 and 0.80) [29].”
MethodInfluentialHotel guests’ perception of best green practices: A content analysis of online reviews
Yi, S., Li, X., & Jai, T.-M. C. (2016) · Tourism and Hospitality Research · doi:10.1177/1467358416637251 · Cited by 174
“This is a more significant measurement method than a simple agreement calculation (Cohen, 1968).”
MethodOn the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation
Wang, Y., Wang, C., Li, R., & Lin, H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.249 · Cited by 214
B7-H3, a potential therapeutic target, is expressed in diffuse intrinsic pontine glioma
Zhou, Z., Luther, N., Ibrahim, G. M., Hawkins, C. E., Vibhakar, R., Handler, M. H., & Souweidane, M. M. (2012) · Journal of Neuro-Oncology · doi:10.1007/s11060-012-1021-2 · Cited by 160
“In the reading of the immunohistochemistry study, weighted kappa-statistic [13] was used to quantify agreement along the categorical scale between the two viewers.”
Execution of UML models: a systematic review of research and practice
Ciccozzi, F., Malavolta, I., & Selić, B. (2018) · Software & Systems Modeling · doi:10.1007/s10270-018-0675-4 · Cited by 168
“As suggested by Wohlin et al. [51], a random sample of 20 potentially relevant research studies and tools was identified and the inter-researcher agreement was measured using the Cohen–Kappa coefficient [12].”
MethodUsing word embeddings in Twitter election classification
Yang, X., Macdonald, C., & Ounis, I. (2017) · Information Retrieval · doi:10.1007/s10791-017-9319-5 · Cited by 186
“Using Cohen’s kappa (Cohen 1968), we found a moderate agreement of 52% between all assessors.”
MethodGalsky, M. D., Daneshmand, S., Izadmehr, S., Gonzalez‐Kozlova, E., Chan, K., Lewis, S. C., Achkar, B. E., Dorff, T. B., Cetnar, J. P., Neil, B. O., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02568-1 · Cited by 173
“Inter-rater agreement was assessed using the weighted kappa statistic 43 .”
MethodKumar, R., Robson, K. M., & Smith, A. M. R. (1984) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(84)90039-4 · Cited by 154
Three months after severe head injury: psychiatric and social impact on relatives.
Livingston, M. G., Brooks, D. N., & Bond, M. (1985) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.48.9.870 · Cited by 153
Kohmer, N., Toptan, T., Pallas, C., Karaca, O., Pfeiffer, A., Westhaus, S., Widera, M., Berger, A., Hoehl, S., Kammel, M., et al. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10020328 · Cited by 152
“The agreement between the Ag-RDTs and the cell culture results was evaluated using Cohen’s weighted kappa index (K value) [21].”
MethodThe impact of domestic violence on mothers' prenatal representations of their infants
Huth‐Bocks, A. C., Levendosky, A. A., Theran, S. A., & Bogat, G. A. (2004) · Infant Mental Health Journal · doi:10.1002/imhj.10094 · Cited by 141
“(Cohen, 1968; Fleiss, Cohen, & Everitt, 1969) for subscales and both percent agreement and Cohen’s # for overall classifications.”
BackgroundHadler-Olsen, S., Sandvik, K., El‐Agroudi, M. A., & Øgaard, B. (2011) · European Journal of Orthodontics · doi:10.1093/ejo/cjr068 · Cited by 153
“64 (Cohen, 1968).”
Motor Impairments in Children with Autism Spectrum Disorder: A Systematic Review and Meta-analysis
Kangarani‐Farahani, M., Malik, M. A., & Zwicker, J. G. (2023) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-023-05948-1 · Cited by 173
“Kappa statistics were used to calculate the level of agreement (Cohen, 1968) at each step and both reviewers agreed on including each paper in the systematic review.”
Barriers to scientific contributions: The author's formula
Armstrong, J. S. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011201 · Cited by 152
Predictors of developmentally appropriate classroom practices in kindergarten through third grade
Maxwell, K. L., McWilliam, R. A., Hemmeter, M. L., Ault, M. J., & Schuster, J. W. (2001) · Early Childhood Research Quarterly · doi:10.1016/s0885-2006(01)00118-1 · Cited by 150
Chapman, B. E., Lee, S., Kang, H. P., & Chapman, W. W. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.03.011 · Cited by 152
“We computed Fleiss’ Kappa to assess the overall multi-reader agreement as well as Cohen’s Weighted Kappa Coefficient [11] with squared weighting for each pair of raters.”
MethodSala, E., Akın, O., Moskowitz, C. S., Eisenberg, H. F., Kuroiwa, K., Ishill, N. M., Rajashanker, B., Scardino, P. T., & Hricak, H. (2006) · Radiology · doi:10.1148/radiol.2383050657 · Cited by 150
Inter-Coder Agreement in Qualitative Coding: Considerations for its Use
Halpin, S. N. (2024) · American Journal of Qualitative Research · doi:10.29333/ajqr/14887 · Cited by 149
TRANSPARENCY AND OTHER JOURNALISTIC NORMS ON TWITTER: The role of gender
Lasorsa, D. (2012) · Journalism Studies · doi:10.1080/1461670x.2012.657909 · Cited by 138
Valla, J., Bergeron, L., Bérubé, H., Gaudet, N., & St‐Georges, M. (1994) · Research on Child and Adolescent Psychopathology · doi:10.1007/bf02168082 · Cited by 175
Barth, M., Capelle, H., Weidauer, S., Weiß, C., Münch, E., Thomé, C., Luecke, T., Schmiedek, P., Kasuya, H., & Vajkoczy, P. (2006) · Stroke · doi:10.1161/01.str.0000254601.74596.0f · Cited by 149
Evaluating neurological outcome measures: the bare essentials.
Hobart, J. C., Lamping, D. L., & Thompson, A. J. (1996) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.60.2.127 · Cited by 149
A checklist to improve reporting of group-based behaviour-change interventions
Borek, A. J., Abraham, C., Smith, J. R., Greaves, C. J., & Tarrant, M. (2015) · BMC Public Health · doi:10.1186/s12889-015-2300-6 · Cited by 135
“Inter-coder reliability has been frequently assessed using Cohen’s kappa [50], which adjusts for the degree of agreement that can be expected to occur by chance alone.”
MethodBerrone, P., Ricart, J. E., Duch, A., Bernardo, V., Salvador, J., Piedra-Peña, J., & Planas, M. R. (2019) · Sustainability · doi:10.3390/su11082339 · Cited by 178
“In Table 4, we present Cohen’s kappa [92,93] and its generalization by Conger [94], along with the Brennan and Prediger coefficient [95], Scott/Fleiss’s kappa/pi [96], Gwet’s AC [89,97], and Krippendorff’s alpha coefficient [98] for the case study of the La Chira WWTP.”
BackgroundMethodLarsson, M., Hedelin, B., Johansson, I. S., & Athlin, E. E. (2005) · Cancer Nursing · doi:10.1097/00002820-200511000-00004 · Cited by 167
“The reliability was studied by means of an observer agreement test with very good result.(29,30) This was performed by the first author and an oncology nurse, both with advanced education in clinical nutrition, and experience from the actual group of patients and their specific problems.”
MethodPfister, D. G., Rubin, D. M., Elkin, E. B., Neill, U. S., Duck, E., Radzyner, M. H., & Bach, P. B. (2015) · JAMA Oncology · doi:10.1001/jamaoncol.2015.3151 · Cited by 148
Lausch, A., Heurich, M., Gordalla, D., Dobner, H. J., Gwillym-Margianto, S., & Salbach, C. (2013) · Forest Ecology and Management · doi:10.1016/j.foreco.2013.07.043 · Cited by 147
Prevalence of self-reported depressive symptoms in young adolescents.
Schoenbach, V. J., Kaplan, B. H., Wagner, E. H., Grimson, R. C., & Miller, F. T. (1983) · American Journal of Public Health · doi:10.2105/ajph.73.11.1281 · Cited by 147
Cervical spine radiculopathy epidemiology: A systematic review
Mansfield, M., Smith, T., Spahr, N., & Thacker, M. (2020) · Musculoskeletal Care · doi:10.1002/msc.1498 · Cited by 131
The Cambridge Handbook of Personality Psychology
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108264822 · Cited by 161
Schilling, E. A., Baucom, D. H., Burnett, C. K., Sandin, E., & Ragland, L. (2003) · Journal of Family Psychology · doi:10.1037/0893-3200.17.1.41 · Cited by 147
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.