Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
Items 801–900 of 1,098
Risks and benefits of speech recognition for clinical documentation: a systematic review
Hodgson, T., & Coiera, E. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv152 · Cited by 102
“Inter-rater agreement was calculated using Cohen’s j.(13) The observed agreement was statistically significant at 95.”
MethodReliability of the Functional Mobility Scale for Children with Cerebral Palsy
Harvey, A. R., Morris, M. E., Graham, H. K., Wolfe, R. S., & Baker, R. J. (2010) · Physical & Occupational Therapy In Pediatrics · doi:10.3109/01942630903454930 · Cited by 113
“Weighted kappa was developed for use with ordinal data (Cohen, 1968) and reflects the degree of disagreement by attaching greater emphasis to large differences between ratings as compared to small differences (Sim & Wright, 2005).”
MethodMcCarty, C. A., Lau, A. S., Valeri, S. M., & Weisz, J. R. (2003) · Research on Child and Adolescent Psychopathology · doi:10.1023/b:jacp.0000007582.61879.6f · Cited by 114
“It is the recommended coefficient of agreement for ordered category data ( Cohen, 1968 ).”
The Intentions of Parents in Influencing the Career Development of Their Children
Young, R. A., & Friesen, J. D. (1992) · The Career Development Quarterly · doi:10.1002/j.2161-0045.1992.tb00326.x · Cited by 155
Classification of Passes in Football Matches Using Spatiotemporal Data
Chawla, S., Estephan, J., Gudmundsson, J., & Horton, M. (2017) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3105576 · Cited by 121
Hasan, S. S., Deng, X., Li, Z., & Chen, D. (2017) · Sustainability · doi:10.3390/su9040505 · Cited by 122
Sleep classification from wrist-worn accelerometer data using random forests
Sundararajan, K., Georgievska, S., te Lindert, B. H. W., Gehrman, P. R., Ramautar, J., Mazzotti, D. R., Sabia, S., Weedon, M. N., van Someren, E. J. W., Ridder, L., et al. (2021) · Scientific Reports · doi:10.1038/s41598-020-79217-x · Cited by 106
INCALZI, R. A., CAPPARELLA, O., GEMMA, A., LANDI, F., BRUNO, E., DI MEO, F., & CARBONIN, P. (1997) · Journal of Internal Medicine · doi:10.1046/j.1365-2796.1997.00132.x · Cited by 112
An index of scientific quality for health reports in the lay press
Oxman, A. D., Guyatt, G. H., Cook, D. J., Jaeschke, R., Heddle, N. M., & Keller, J. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90166-x · Cited by 130
MGMT expression predicts response to temozolomide in pancreatic neuroendocrine tumors
Cros, J. F., Hentic, O., Rebours, V., Zappa, M., Gille, N., Théou–Anton, N., Vernerey, D., Maire, F., Lévy, P., Bédossa, P., et al. (2016) · Endocrine Related Cancer · doi:10.1530/erc-16-0117 · Cited by 112
Conceptual Issues in Research on Stress and Health
Semmer, N. K., McGrath, J. E., & Beehr, T. A. (2004) · CRC Press eBooks · doi:10.1201/9781420039702.ch1 · Cited by 141
The measurement of interexaminer agreement on periodontal disease
Fleiss, J. L., & Chilton, N. W. (1983) · Journal of Periodontal Research · doi:10.1111/j.1600-0765.1983.tb00397.x · Cited by 111
Agreement between physicians on assessment of outcome following severe head injury
Maas, A. I. R., Braakman, R., Schouten, H. J. A., Minderhoud, J. M., & van Zomeren, A. H. (1983) · Journal of Neurosurgery · doi:10.3171/jns.1983.58.3.0321 · Cited by 113
Measuring the potency of risk factors for clinical or policy significance.
KRAEMER, H. C., Kazdin, A. E., Offord, D. R., Kessler, R. C., Jensen, P. S., & Kupfer, D. J. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.3.257 · Cited by 117
DISCUSSION PROCEDURES AND DECISION-MAKING PERFORMANCE.: A Test of a Functional Perspective
HIROKAWA, R. Y. (1985) · Human Communication Research · doi:10.1111/j.1468-2958.1985.tb00073.x · Cited by 86
Overview of the 8th Workshop on Asian Translation
Nakazawa, T., Nakayama, H., Ding, C., Dabre, R., Higashiyama, S., Mino, H., Goto, I., Pa, W. P., Kunchukuttan, A., Parida, S., et al. (2021) · Workshop on Asian Translation (WAT) · doi:10.18653/v1/2021.wat-1.1 · Cited by 111
“The weights for the weighted κ (Cohen, 1968) is defined as |Evaluation1 − Evaluation2|/4.”
BackgroundFRODI, A. M., & Thompson, R. A. (1985) · Child Development · doi:10.1111/j.1467-8624.1985.tb00196.x · Cited by 111
Hutchinson, T. A. (1979) · JAMA · doi:10.1001/jama.242.7.633 · Cited by 108
A note on the linearly weighted kappa coefficient for ordinal scales
Vanbelle, S., & Albert, A. (2008) · Statistical Methodology · doi:10.1016/j.stamet.2008.06.001 · Cited by 110
“Cohen (1968) showed that,25 under specific conditions, the weighted kappa coefficient is equivalent to the26 product-moment correlation coefficient.”
BackgroundMethodSarcoidosis: assessment of disease severity using HRCT
Drent, M., Vries, J. D., Lenters, M., Lamers, R. J. S., Kos, S., Wouters, E. F. �., van Dieijen‐Visser, M. P., & Verschakelen, J. (2003) · European Radiology · doi:10.1007/s00330-003-1965-x · Cited by 110
“00 excellent agreement [27].”
Screening and Follow-Up of Living Kidney Donors: A Systematic Review of Clinical Practice Guidelines
Tong, A., Chapman, J. R., Wong, G., de Bruijn, J., & Craig, J. C. (2011) · Transplantation · doi:10.1097/tp.0b013e3182328276 · Cited by 110
Pollock, T., Whitbred, R., & Contractor, N. (2000) · Human Communication Research · doi:10.1111/j.1468-2958.2000.tb00759.x · Cited by 132
The inter- and intra-observer reliability of a locomotion scoring scale for sheep
Kaler, J., Wassink, G. J., & Green, L. E. (2008) · The Veterinary Journal · doi:10.1016/j.tvjl.2007.12.028 · Cited by 111
Relevance Vector Machine Learning for Neonate Pain Intensity Assessment Using Digital Imaging
Gholami, B., Haddad, W. M., & Tannenbaum, A. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2009.2039214 · Cited by 114
Koletzko, S., Jesch, I., Faus-Keler, T., Briner, J., Meier‐Ruge, W., Müntefering, H., Coerdt, W., Wessel, L., Keller, K. M., Nützenadel, W., et al. (1999) · Gut · doi:10.1136/gut.44.6.853 · Cited by 110
Statistical Modeling of Expert Ratings on Medical Treatment Appropriateness
Uebersax, J. S. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476291 · Cited by 116
Herting, M. M., Uban, K. A., Gonzalez, M. R., Baker, F. C., Kan, E., Thompson, W. K., Granger, D. A., Albaugh, M. D., Anokhin, A. P., Bagot, K. S., et al. (2021) · Frontiers in Endocrinology · doi:10.3389/fendo.2020.549928 · Cited by 130
“The polychoric correlation is useful for examining agreement between ordered category data (53), whereas k is weighted to consider random chance of agreement (54).”
MethodRobust Detection of COVID-19 in Cough Sounds: Using Recurrence Dynamics and Variable Markov Model
Mouawad, P., Dubnov, T., & Dubnov, S. (2021) · SN Computer Science · doi:10.1007/s42979-020-00422-6 · Cited by 116
Mental Load and Fatigue Assessment Instruments: A Systematic Review
García, J. D., González‐Ponce, I., Ponce-Bordón, J. C., López-Gajardo, M. Á., Bravo, I. R., Rubio‐Morales, A., & García‐Calvo, T. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19010419 · Cited by 122
Regnell, B., Runeson, P., & Thelin, T. (2000) · Empirical Software Engineering · doi:10.1023/a:1009848320066 · Cited by 119
Kappa coefficients in epidemiology: An appraisal of a reappraisal
Kraemer, H. C., & Bloch, D. A. (1988) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(88)90032-7 · Cited by 108
Correlation coefficients in medical research: from product moment correlation to the odds ratio
Kraemer, H. C. (2006) · Statistical Methods in Medical Research · doi:10.1177/0962280206070650 · Cited by 105
McColl, E., Junghard, O., Wiklund, I., & Revicki, D. A. (2005) · The American Journal of Gastroenterology · doi:10.1111/j.1572-0241.2005.40945.x · Cited by 110
Assessment of Adenoidal Obstruction in Children: Clinical Signs Versus Roentgenographic Findings
Paradise, J. L., Bernard, B. S., Colborn, D. K., & Janosky, J. E. (1998) · Pediatrics · doi:10.1542/peds.101.6.979 · Cited by 108
Kelley, G. A., & Kelley, K. S. (2010) · Journal of Hypertension · doi:10.1097/hjh.0b013e3283357d16 · Cited by 105
Competing with New Product Technologies: A Process Model of Strategy
Das, S. S., & Van de Ven, A. H. (2000) · Management Science · doi:10.1287/mnsc.46.10.1300.12276 · Cited by 105
Global coherence in younger and older adults: Influence of cognitive processes and discourse type
Wright, H. H., Koutsoftas, A. D., Capilouto, G. J., & Fergadiotis, G. (2013) · Aging Neuropsychology and Cognition · doi:10.1080/13825585.2013.794894 · Cited by 121
“Second, Cohen’s kappa (Cohen, 1968) was also computed.”
Reliability of Concussion History in Former Professional Football Players
KERR, Z. Y., MARSHALL, S. W., & GUSKIEWICZ, K. M. (2012) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0b013e31823240f2 · Cited by 105
Mei, F., Gao, Q., Chen, F., Zhao, L., Shang, Y., Hu, K., Zhang, W., Zhao, B. Q., & Ma, B. (2020) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2020.09.033 · Cited by 114
The Operationalized Psychodynamic Diagnostics System: Clinical Relevance, Reliability and Validity
Cierpka, M., Grande, T., Rudolf, G., von der Tann, M., & Stasch, M. (2007) · Psychopathology · doi:10.1159/000101363 · Cited by 108
Early Interests and Joint Engagement in Typical Development, Autism, and Down Syndrome
Adamson, L. B., Deckner, D. F., & Bakeman, R. A. (2009) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-009-0914-1 · Cited by 120
“Reliability for the interest ratings was assessed with weighted kappa (Bakeman and Gottman 1997; Cohen 1968).”
MethodFink, A., Morton, S. C., Beck, J. C., Hays, R. D., Spritzer, K., Oishi, S., & Moore, A. A. (2002) · Journal of the American Geriatrics Society · doi:10.1046/j.1532-5415.2002.50467.x · Cited by 105
Classification of Crop Yield Variability in Irrigated Production Fields
Dobermann, A., Ping, J. L., Adamchuk, V. I., Simbahan, G. C., & Ferguson, R. B. (2003) · Agronomy Journal · doi:10.2134/agronj2003.1105 · Cited by 108
“…of Yield Classification Results agreement for categorical data (Kw), which was defined by Several statistics were computed to assess the results of Cohen (1968) as different yield classification procedures in terms of (i) yield variance accounted for and (ii) spatial agreement between two maps…”
MethodReliability of Scores on the Stroke Rehabilitation Assessment of Movement (STREAM) Measure
Daley, K., Mayo, N., & Wood-Dauphinée, S. (1999) · Physical Therapy · doi:10.1093/ptj/79.1.8 · Cited by 107
Peer interactions of autistic children and adolescents
Lord, C., & MaGill-Evans, J. (1995) · Development and Psychopathology · doi:10.1017/s095457940000674x · Cited by 98
Conflictive Engagement, Positive Affect, and Menarche in Families with Seventh-Grade Girls
Holmbeck, G. N., & Hill, J. P. (1991) · Child Development · doi:10.1111/j.1467-8624.1991.tb01587.x · Cited by 107
Current State of the Art for Statistical Modelling of Species Distributions
Hegel, T. M., Cushman, S. A., Evans, J. S., & Huettmann, F. (2009) · Springer eBooks · doi:10.1007/978-4-431-87771-4_16 · Cited by 113
“To overcome limitations in the Kappa a weighting function (Cohen 1968) has been implemented to account for near agreement and adjust for expectation in the frequency of observations.”
MethodThe reliability of the Canadian triage and acuity scale: Meta-analysis
Mirhaghi, A., Ebrahimi, M., Heydari, A., & Mazlom, R. (2015) · North American Journal of Medical Sciences · doi:10.4103/1947-2714.161243 · Cited by 121
“North American Journal of Medical Sciences | July 2015 | Volume 7 | Issue 7 | 304 higher reliability than un-weighted kappa coeffi cient because it put more emphasis on the large differences between ratings than to small differences.[36] Thus It`s worth-mentioning even one category difference in allocating patients in…”
BackgroundInterrater agreement statistics with skewed data: Evaluation of alternatives to Cohen’s kappa.
Xu, S., & Lorber, M. F. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0037489 · Cited by 103
Weatherson, K. A., McKay, R. C., Gainforth, H. L., & Jung, M. E. (2017) · BMC Public Health · doi:10.1186/s12889-017-4846-y · Cited by 118
“Percent agreements, Cohen’s Kappa statistic [39] and prevalence-adjusted bias-adjusted Kappa statistic (i.”
Penteriani, V., Zarzo‐Arias, A., Novo‐Fernández, A., Bombieri, G., & López-Sánchez, C. A. (2019) · Global Change Biology · doi:10.1111/gcb.14564 · Cited by 109
Beyond alcoholism: identifying older, at-risk drinkers in primary care.
Moore, A. A., Beck, J. C., Babor, T. F., Hays, R. D., & Reuben, D. B. (2002) · Journal of Studies on Alcohol · doi:10.15288/jsa.2002.63.316 · Cited by 105
Evaluation of the consistency of wine quality assessments from expert wine tasters
Gawel, R., & Godden, P. W. (2008) · Australian Journal of Grape and Wine Research · doi:10.1111/j.1755-0238.2008.00001.x · Cited by 107
Yee, C., Farewell, V. T., Isenberg, D. A., Prabu, A., Sokoll, K. B., Teh, L., Rahman, A., Bruce, I. N., Griffiths, B., Akil, M., et al. (2006) · Arthritis & Rheumatism · doi:10.1002/art.22162 · Cited by 107
van der Mei, I. A. F., Blizzard, L., Ponsonby, A., & Dwyer, T. (2006) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-05-0969 · Cited by 107
“The proportion in exact agreement (percent agreement) and the weighted Cohen’s n statistic (28) were calculated as measures of agreement (29).”
MethodValidity and Reliability of Subject and Mother Reporting of Perinatal Factors
Sanderson, M., Williams, M. A., White, E., Daling, J. R., Holt, V. L., Malone, K. E., Self, S. G., & Moore, D. E. (1998) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009425 · Cited by 107
“The agreement was summarized as the bias, measured as the mean difference in category between subject (or mother) response and birth certificate, and the precision, measured as the Spearman correlation coefficient or weighted kappa ( 11 ) between the category of subject (or mother response) and birth certificate (10).”
MethodEnvironmental "White Noise": An Intervention for Verbally Agitated Nursing Home Residents
Burgio, L., Scilley, K., Hardin, J. M., Hsu, C., & Yancey, J. (1996) · The Journals of Gerontology Series B: Psychological Sciences and Social Sciences · doi:10.1093/geronb/51b.6.p364 · Cited by 107
The reliability of the ?present state examination?
Kendell, R. E., Everett, B., Cooper, J. E., Sartorius, N., & DAVID, M. E. (1968) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/bf00577836 · Cited by 107
Does it matter whom and how you ask? Inter- and intra-rater agreement in the Ontario Health Survey
Grootendorst, P. V., Feeny, D. H., & Furlong, W. J. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00314-9 · Cited by 107
Farmer, A. J., McSharry, J., Rowbotham, S., McGowan, L., Ricci‐Cabello, I., & French, D. P. (2016) · Diabetic Medicine · doi:10.1111/dme.12987 · Cited by 106
Towards a Better Metric for Evaluating Question Generation Systems
Nema, P., & Khapra, M. M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1429 · Cited by 106
“In Table 3, we report the average inter-annotator agreement between the ratings using Cohen’s kappa ( κ ) score (Cohen, 1968).”
MethodKelley, G. A., Kelley, K. S., & Franklin, B. (2006) · Journal of Cardiopulmonary Rehabilitation · doi:10.1097/00008483-200605000-00002 · Cited by 106
Longitudinal Associations Among Peer Victimization and Physical and Mental Health Problems
Biebl, S. J. W., DiLalla, L. F., Davis, E. K., Lynch, K. A., & Shinn, S. O. (2011) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsr025 · Cited by 99
“Inter-rater reliability was assessed using a weighted (Cohen, 1968), a conservative estimate used when categories have a disproportionately high or low frequency, as was the case here because bullying was relatively infrequent.”
BackgroundMethodVasilevskis, E. E., Morandi, A., Boehm, L. M., Pandharipande, P. P., Girard, T. D., Jackson, J. C., Thompson, J. L., Shintani, A., Gordon, S. M., Pun, B. T., et al. (2011) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2011.03673.x · Cited by 110
“To determine agreement between bedside and research nurses’ assessments of the CAM-ICU and RASS, assessments that were no more than 4 hours apart were paired, and weighted kappa statistics were calculated.(42,43) When multiple bedside assessments were available within the given time frame, the bedside nurse assessment…”
MethodAn empirically evaluated checklist for surveys in software engineering
Molléri, J. S., Petersen, K., & Mendes, E. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106240 · Cited by 134
The Effect of the Traditional Mediterranean-Style Diet on Metabolic Risk Factors: A Meta-Analysis
Garcia, M., Bihuniak, J., Shook, J., Kenny, A., Kerstetter, J., & Huedo-Medina, T. (2016) · Nutrients · doi:10.3390/nu8030168 · Cited by 100
“The kappa (κ) coefficient was used to calculate categorical agreement [12] (kappa = 0.”
MethodKelley, G. A., & Kelley, K. S. (2007) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2006.04.019 · Cited by 73
“Cohen's kappa [19] for inter-rater agreement prior to correcting discrepant items was 0.”
BackgroundInconsistency indices for pairwise comparison matrices: a numerical study
Brunelli, M., Canal, L., & Fedrizzi, M. (2013) · Annals of Operations Research · doi:10.1007/s10479-013-1329-0 · Cited by 105
“4.2, we computed the Spearman Coefficient (23) for all (10 2 ) pairs of indices {Ii, Ij }, i = 1, . . . ,9; j = i, . . . ,10.”
MethodInfluentialHuman fetal and neonatal movement patterns: Gender differences and fetal‐to‐neonatal continuity
Almli, C. R., Ball, R. H., & Wheeler, M. E. (2001) · Developmental Psychobiology · doi:10.1002/dev.1019 · Cited by 105
“97, indicating high inter-scorer and intra-scorer agreement (Cohen, 1968).”
BackgroundInfluentialAdult Personality Functioning Assessment (APFA): An Investigator-Based Standardised Interview
Hill, J., Harrington, R., Fudge, H., Rutter, M., & Pickles, A. (1989) · British Journal of Psychiatry · doi:10.1192/bjp.155.1.24 · Cited by 103
“For thesecategoricalratings, inter-rater agreement was assessedusing the kappa statistic (Cohen, 1968).”
MethodValue of MRI and diffusion-weighted MRI for the diagnosis of locally recurrent rectal cancer
Lambregts, D. M. J., Cappendijk, V. C., Maas, M., Beets, G. L., & Beets‐Tan, R. G. H. (2011) · European Radiology · doi:10.1007/s00330-010-2052-8 · Cited by 105
“00 excellent interobserver agreement) were calculated to determine interobserver agreement between the two readers [25].”
MethodRohrbach, L. A., Dent, C. W., Skara, S. N., Sun, P., & Sussman, S. Y. (2006) · Prevention Science · doi:10.1007/s11121-006-0056-z · Cited by 119
“For the dichotomous variable, a weighted kappa was calculated (Cohen, 1968).”
MethodJung, S. I., Donati, O. F., Vargas, H. A., Goldman, D., Hricak, H., & Akın, O. (2013) · Radiology · doi:10.1148/radiol.13130029 · Cited by 107
The Baltic Sea scale inventory of benthic faunal communities
Gogina, M., Nygård, H., Blomqvist, M., Daunys, D., Josefson, A. B., Kotta, J., Maximov, A. A., Warzocha, J., Yermakov, V., Gräwe, U., et al. (2016) · ICES Journal of Marine Science · doi:10.1093/icesjms/fsv265 · Cited by 112
“Additionally, we applied Cohen’s (1968) k coefficient of agreement calculated using “psych” R library (Revelle, 2015).”
MethodA Survey of Industrial Control System Testbeds
Holm, H., Karresand, M., Vidström, A., & Westring, E. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-26502-5_2 · Cited by 131
“Redundant judgments were used to enable measuring the group’s internal agreement with the statistical metric Cohen’s Kappa [10].”
MethodHolman, D. J., Escaffi‐Schwarz, M., Vasquez, C., Irmer, J. P., & Zapf, D. (2023) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12450 · Cited by 124
Acoustic/prosodic and lexical correlates of charismatic speech
Rosenberg, A., & Hirschberg, J. (2005) · Interspeech (USB) · doi:10.21437/interspeech.2005-329 · Cited by 105
“We used the weighted kappa statistic [4] with quadratic weighting to determine the inter-subject agreement.”
MethodGrotevant, H. D., Rueter, M. A., Korff, L. V., & Gonzalez, C. J. (2010) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2010.02330.x · Cited by 116
“Journal of Child Psychology and Psychiatry 2010 Association for Child and Adolescent Mental Health. degree of disagreement between coders (Cohen, 1968).”
BackgroundUse of Google Street View to Assess Environmental Contributions to Pedestrian Injury
Mooney, S. J., DiMaggio, C. J., Lovasi, G. S., Neckerman, K. M., Bader, M. D. M., Teitler, J. O., Sheehan, D. M., Jack, D. W., & Rundle, A. (2016) · American Journal of Public Health · doi:10.2105/ajph.2015.302978 · Cited by 105
Ward, W. T., Vogt, M. T., Grudziak, J. S., Tümer, Y., Cook, P. C., & Fitch, R. D. (1997) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-199705000-00004 · Cited by 105
Malevolent object representations in borderline personality disorder and major depression.
Nigg, J. T., Lohr, N. E., Westen, D., & Gold, L. J. (1992) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.101.1.61 · Cited by 104
Arsenijevic, J., Pavlova, M., Rechel, B., & Groot, W. (2016) · PLOS ONE · doi:10.1371/journal.pone.0157765 · Cited by 96
Jackson, K. M., & Sher, K. J. (2005) · Psychology of Addictive Behaviors · doi:10.1037/0893-164x.19.4.339 · Cited by 103
“Note that these “weighted” kappas are not weighted kappas as the term is traditionally used in the literature to describe a kappa variant that takes into account gradation in level of disagreement (Cohen, 1968).”
BackgroundPerformance and Repeatability of the NEI-VFQ-25 in Patients With Dry Eye
Nichols, K. K., Mitchell, G. L., & Zadnik, K. (2002) · Cornea · doi:10.1097/00003226-200208000-00009 · Cited by 114
“94 (17) Accomplish less because of vision 0.”
BackgroundJones, K. L., Robinson, L. K., Bakhireva, L. N., Marintcheva, G., Storojev, V., Strahova, A., Sergeevskaya, S., Budantseva, S., Mattson, S. N., Riley, E. P., et al. (2006) · PEDIATRICS · doi:10.1542/peds.2006-1037 · Cited by 104
Establishing Clinically Relevant Standards for Tachypnea in Febrile Children Younger Than 2 Years
Taylor, J. A. (1995) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpedi.1995.02170150063011 · Cited by 104
Demerouti, E., Verbeke, W. J. M. I., & Bakker, A. B. (2005) · Journal of Management · doi:10.1177/0149206304271602 · Cited by 94
“In both validation samples, the highest Kappa resulted for the three-cluster solution, followed by the five-cluster solution.”
MethodInfluentialTelepathology diagnosis by means of digital still images: An international validation study
Weinberg, D. S., Allaert, F., Dusserre, P., Drouot, F., Retailliau, B., Welch, W. R., Longtine, J., BRODSKY, G. L., Folkerth, R. D., & Doolittle, M. H. (1996) · Human Pathology · doi:10.1016/s0046-8177(96)90363-9 · Cited by 104
Hope, T., & Fairburn, C. G. (1992) · Psychological Medicine · doi:10.1017/s0033291700032888 · Cited by 104
Specificity of the clinical diagnosis of dementia
Todorov, A. B., Go, R. C. P., Constantinidis, J., & Elston, R. C. (1975) · Journal of the Neurological Sciences · doi:10.1016/0022-510x(75)90116-1 · Cited by 104
An Informal Science Education Program’s Impact on STEM Major and STEM Career Outcomes
Habig, B., Gupta, P., Levine, B., & Adams, J. (2020) · Research in Science Education · doi:10.1007/s11165-018-9722-y · Cited by 103
“To assess consistency across coders (reliability), we calculated inter-coder agreement using both percent agreement (Lombard et al. 2002) and Cohen’s kappa (Cohen 1968) applying methodologies established for semi-structured interviews (Campbell et al. 2013).”
BackgroundMethodInfluentialDefects of the Lamina Cribrosa in Eyes with Localized Retinal Nerve Fiber Layer Loss
Tatham, A. J., Miki, A., Weinreb, R. N., Zangwill, L. M., & Medeiros, F. A. (2013) · Ophthalmology · doi:10.1016/j.ophtha.2013.08.018 · Cited by 108
Prospective sampling based on model ensembles improves the detection of rare species
Lay, G. L., Engler, R., Franc, E., & Guisan, A. (2010) · Ecography · doi:10.1111/j.1600-0587.2010.06338.x · Cited by 118
“For the GAM-based projections, the threshold between the two classes was chosen to maximize the weighted Kappa (Cohen 1968).”
MethodComparing two versions of the Karolinska Sleepiness Scale (KSS)
Miley-Åkerstedt, A., Kecklund, G., & Åkerstedt, T. (2016) · Sleep and Biological Rhythms · doi:10.1007/s41105-016-0048-8 · Cited by 103
“However, since the KSS scale is ordered, a more appropriate method is to use Cohen’s weighted Kappa [12] which takes into account how close the rating in one version is to the rating in the other version.”
MethodWhen does Relevance Mean Usefulness and User Satisfaction in Web Search?
Mao, J., Liu, Y., Zhou, K., Nie, J.-Y., Song, J., Zhang, M., Ma, S., Sun, J., & Luo, H. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911507 · Cited by 96
“To study the correlation between Uu and R, we compute Pearson’s correlation coefficient r and Cohen’s Weighted κ between these two document-level measures.”
MethodInfluentialOlsen, C. F., & Bergland, A. (2017) · BMC Geriatrics · doi:10.1186/s12877-017-0514-4 · Cited by 108
“The weighted κ score measures the agreement of test–retest, adjusted for the amount of agreement expected by chance and the magnitude of disagreement [32].”
BackgroundJousse‐Joulin, S., Nowak, E., Cornec, D., Brown, J. E., Carr, A., Carotti, M., Fisher, B. A., Fradin, J. M., Hočevar, A., Jonsson, M. V., et al. (2017) · RMD Open · doi:10.1136/rmdopen-2016-000364 · Cited by 111
Ende-Verhaar, Y. M., Meijboom, L. J., Kroft, L. J. M., Beenen, L. F. M., Boon, G. J. A. M., Middeldorp, S., Nossent, E. J., Symersky, P., Huisman, M. V., Bogaard, H. J., et al. (2019) · The Journal of Heart and Lung Transplantation · doi:10.1016/j.healun.2019.03.003 · Cited by 103
Medical Diagnostic Tests: A Review of Test Anatomy, Phases, and Statistical Treatment of Data
Bolboaca, S. D. (2019) · Computational and Mathematical Methods in Medicine · doi:10.1155/2019/1891569 · Cited by 108
“Different weights reflecting the importance of agreement and the weights (linear, proportional to the number of categories apart or quadratic, proportional to the square of the number of classes apart) must be established by the researcher [131].”
BackgroundLassalle, G., Ferreira, M. P., Rosa, L. E. C. L., & Filho, C. R. D. S. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.05.002 · Cited by 103
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.