Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (Kw) is presented. The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, (3) implies that Cohen's quadratically weighted kappa reduces to the Pearson product-moment correlation 𝜌 1,2 if we impose equal means and variances, that is, 𝜇 1 = 𝜇 2 and 𝜎 1 2 = 𝜎 2 2 (Cohen, 1968).”
From Quadratically Weighted Agreement Coefficients: Interpretations and Connections (2026) · cites this paper as Cohen, 1968
In the citing paper
“Given the ordinal nature of the ratings, inter-rater agreement was evaluated using Cohen's quadratic weighted κ at the item level (Cohen, 1968), while intraclass correlation coefficients (ICCs) were used at the total score level.”
From Measuring mindfulness: A systematic review and evaluation of self-report mindfulness scales (2026) · cites this paper as Cohen, 1968
In the citing paper
“PABAK was used rather than Cohen's kappa statistic 101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent).”
In the citing paper
“Prevalence-and bias-adjusted kappa was used rather than Cohen's kappa statistic.101”
In the citing paper
“Inter-rater reliability analysis (two raters; a linearly weighted kappa analysis, κw)18 applied to our ordinal scale data revealed a very good reliability for the scoring procedure, κw = 0.844, 95% CI [0.774, 0.914]19.”
From The early effects of external and internal strategies on working memory updating training (2018) · cites this paper as [18]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Self-citationTorgerson, W. S. (1958) · Medical Entomology and Zoology · Cited by 2,920
Gulliksen, H. (1950) · John Wiley & Sons, Inc. eBooks · doi:10.1037/13240-000 · Cited by 2,177
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Psychological statistics, 2nd ed.
McNemar, Q. (1955) · Cited by 559
Quantification of Agreement in Psychiatric Diagnosis: A New Approach
Spitzer, R. L. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730250085012 · Cited by 333
Problems and methods of psychophysics.
Stevens, S. S. (1958) · Psychological Bulletin · doi:10.1037/h0044251 · Cited by 258
Foundations of the Theory of Prediction
Glass, G. V., & Rozeboom, W. W. (1967) · American Educational Research Journal · doi:10.2307/1161775 · Cited by 157
Note on Rank Biserial Correlation
Glass, G. V. (1966) · Educational and Psychological Measurement · doi:10.1177/001316446602600307 · Cited by 102
An aid in the computation of correlations based on Q sorts.
Cohen, J. (1957) · Psychological Bulletin · doi:10.1037/h0046220 · Cited by 24
Self-citation
Cited by
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 23,868
Lord, C., Rutter, M., & Le Couteur, A. (1994) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02172145 · Cited by 9,065
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Predictive habitat distribution models in ecology
Guisan, A., & Zimmermann, N. E. (2000) · Ecological Modelling · doi:10.1016/s0304-3800(00)00354-9 · Cited by 7,359
“Examples of such measures are g (Goodman and Kruskal, 1979), Dxy of Somer, Dxy of Kim, Dxy of Wilson (Gonzalez and Nelson, 1996; Guisan and Harrell (2000) for an application in ecology) or the weighted k (Cohen, 1968; Naesset, 1996).”
Background(2005) · PubMed · Cited by 7,011
Interobserver agreement for the assessment of handicap in stroke patients
van Swieten, J. C., Koudstaal, P. J., Visser, M. C., Schouten, H. J., & van Gijn, J. (1988) · Stroke · doi:10.1161/01.str.19.5.604 · Cited by 6,512
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
(2004) · Cited by 5,198
Likert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850
“Cohen (1968) later generalized the formulation to ‘‘weighted kappa’’, to be used with ordinal data such as Likert scales, where the data would be displayed as agreement in a 7 9 7 matrix.”
MethodSoilGrids250m: Global gridded soil information based on machine learning
Hengl, T., Mendes de Jesus, J., Heuvelink, G. B. M., Ruiperez Gonzalez, M., Kilibarda, M., Blagotić, A., Shangguan, W., Wright, M. N., Geng, X., Bauer-Marschallinger, B., et al. (2017) · PLOS ONE · doi:10.1371/journal.pone.0169748 · Cited by 4,733
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Measurement of patient outcome in arthritis
Fries, J. F., Spitz, P., Kraines, R. G., & Holman, H. R. (1980) · Arthritis & Rheumatism · doi:10.1002/art.1780230202 · Cited by 4,382
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884
“Cohen (1968) provides an alternative weighted kappa that allows researchers to differentially penalize disagreements based on the magnitude of the disagreement.”
BackgroundFleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
“For appraising reliability, for example, a useful measure of inter-rater agreement for categorical scales is provided by kappa (Cohen, 1960) or weighted kappa (Spitzer, Cohen, Fleiss and Endicott, 1967; Cohen, 1968a).”
BackgroundMethodInfluentialSelf-citationCharnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
The Non-Hodgkin's Lymphoma Pathologic Classification Project (1982) · Cancer · doi:10.1002/1097-0142(19820515)49:10<2112::aid-cncr2820491024>3.0.co;2-2 · Cited by 3,059
Mutagens, carcinogens, and tumor promoters in our daily food
Sügimura, T. (1982) · Cancer · doi:10.1002/1097-0142(19820515)49:10<1970::aid-cncr2820491005>3.0.co;2-f · Cited by 232
Cornell scale for depression in dementia
Alexopoulos, G. S., Abrams, R. C., Young, R. C., & Shamoian, C. A. (1988) · Biological Psychiatry · doi:10.1016/0006-3223(88)90038-8 · Cited by 2,678
Evaluating resource selection functions
Boyce, M. S., Vernier, P. R., Nielsen, S. E., & Schmiegelow, F. K. A. (2002) · Ecological Modelling · doi:10.1016/s0304-3800(02)00200-4 · Cited by 2,566
“Under situations where one might consider some errors in the classification table to be less or more important, one can use a weighted Kappa, controlling the seriousness of each possible disagreement (Cohen, 1968).”
Hoy, D., Brooks, P., Woolf, A., Blyth, F., March, L., Bain, C., Baker, P., Smith, E., & Buchbinder, R. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.11.014 · Cited by 2,555
“For assessment of the interrater agreement for the summary item, we weighted the Kappa statistic to take ordering of response options into account [20].”
MethodMachine learning applications in genetics and genomics
Libbrecht, M. W., & Noble, W. S. (2015) · Nature Reviews Genetics · doi:10.1038/nrg3920 · Cited by 2,084
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
(1991) · Ophthalmology · doi:10.1016/s0161-6420(13)38012-9 · Cited by 2,048
Consensus development methods, and their use in clinical guideline development.
Murphy, Black, Lamping, McKee, Sanderson, Askham, & Marteau (1998) · Health Technology Assessment · doi:10.3310/hta2030 · Cited by 2,051
Lord, C., Rutter, M., Goode, S., Heemsbergen, J., Jordan, H., Mawhood, L., & Schopler, E. (1989) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02211841 · Cited by 2,030
Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755
“For each re-viewer, the data set extracted contained three quality ratings for each review, yielding a total of six ratings per review.”
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,570
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
“A weighted variant of Cohen’s κ is presented in Cohen (1968).”
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationIsmail, A. I., Sohn, W., Tellez, M., Amaya, A., Sen, A., Hasson, H., & Pitts, N. B. (2007) · Community Dentistry and Oral Epidemiology · doi:10.1111/j.1600-0528.2007.00347.x · Cited by 1,535
McGuffin, P. (1991) · Archives of General Psychiatry · doi:10.1001/archpsyc.1991.01810320088015 · Cited by 1,483
Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228
“2 w.r.t. the other respondents, which is a standard measure for inter-rater agreement (Cohen, 1968).”
BackgroundTrouillas, P., Takayanagi, T., Hallett, M., Currier, R. D., Subramony, S. H., Wessel, K., Bryer, A., Diener, H. C., Massaquoi, S., Gomez, C. M., et al. (1997) · Journal of the Neurological Sciences · doi:10.1016/s0022-510x(96)00231-6 · Cited by 1,263
(2011) · Amsterdam UMC (VU Amsterdam) - Institutional Repository · Cited by 1,209
MEDIA LEGITIMATION EFFECTS IN THE MARKET FOR INITIAL PUBLIC OFFERINGS.
Pollock, T. G., & Rindova, V. P. (2003) · Academy of Management Journal · doi:10.2307/30040654 · Cited by 1,122
“Cohen and Cohen (1983) suggested that one method for dealing with multicollinearity in polynomial regression equations—that is, regressions that include higher-order terms—is to orthogonalize the collinear variables by “partialing out” the common variance.”
MethodPeer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079
Stemler, S. E. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/96jp-xz07 · Cited by 992
“A third consensus estimate of interrater reliability is Cohen’s kappa statistic (Cohen, 1960, 1968).”
Best Practices in Quantitative Methods
Osborne, J. (2008) · Sage eBooks · doi:10.4135/9781412995627 · Cited by 942
(1999) · doi:10.1016/s0090-4295(99)00099-0 · Cited by 1,051
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-61 · Cited by 1,041
“Because this assessment results in dichotomous outcomes, Cohen’s Kappa [2,3] is commonly used to assess the reliability of raters.”
MethodAutism diagnostic interview: A standardized investigator-based instrument
Le Couteur, A., Rutter, M., Lord, C., Rios, P., Robertson, S., Holdgrafer, M., & McLennan, J. (1989) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02212936 · Cited by 1,036
Nutritional Assessment: A Comparison of Clinical Judgment and Objective Measurements
Baker, J. P., Detsky, A. S., Wesson, D. E., Wolman, S. L., Stewart, S., Whitewell, J., Langer, B., & Jeejeebhoy, K. N. (1982) · New England Journal of Medicine · doi:10.1056/nejm198204223061606 · Cited by 1,019
A new prognostic index (MIPI) for patients with advanced-stage mantle cell lymphoma
Hoster, E., Dreyling, M., Klapper, W., Gisselbrecht, C., van Hoof, A., Kluin-Nelemans, H. C., Pfreundschuh, M., Reiser, M., Metzner, B., Einsele, H., et al. (2008) · Blood · doi:10.1182/blood-2007-06-095331 · Cited by 984
“Median platelet count, 109/L (range) 188 (3-1346) 452”
Statistical methods for assessing observer variability in clinical measures.
Brennan, P., & Silman, A. (1992) · BMJ · doi:10.1136/bmj.304.6840.1491 · Cited by 985
Interrater Reliability of the NIH Stroke Scale
Goldstein, L. B., Bertels, C., & Davis, J. N. (1989) · Archives of Neurology · doi:10.1001/archneur.1989.00520420080026 · Cited by 986
Seror, R., Ravaud, P., Bowman, S. J., Baron, G., Tzioufas, A., Theander, E., Gottenberg, J.-E., Bootsma, H., Mariette, X., & Vitali, C. (2010) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2009.110619 · Cited by 965
“Reliability of disease activity scoring The evaluation of clinical vignettes common to 2 raters was used to assess inter-rater reliability: - For the 0-10 PhGA: intraclass correlation coefficient (ICC) and Bland and Altman graphical analysis[23, 24] - For the 5-point scale: global agreement and Kappa statistics[25, 26…”
MethodBeyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976
Goldman, L., Hashimoto, B., Cook, E. F., & Loscalzo, A. (1981) · Circulation · doi:10.1161/01.cir.64.6.1227 · Cited by 978
Autism From 2 to 9 Years of Age
Lord, C., Risi, S., DiLavore, P. S., Shulman, C., Thurm, A., & Pickles, A. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.6.694 · Cited by 973
“Agreementamongcontemporaneous diagnostic measures and between baseline and follow-updiagnosiswasassessedusing statistics that correct for chance agreement for nominal measures.(34) Prediction of autism and ASD used logistic regression.”
BackgroundLoughrey, D. G., Kelly, M. E., Kelley, G. A., Brennan, S., & Lawlor, B. A. (2018) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2017.2513 · Cited by 961
Evolutionary Personality Psychology
Buss, D. M. (1991) · Annual Review of Psychology · doi:10.1146/annurev.ps.42.020191.002331 · Cited by 930
Method agreement analysis: A review of correct methodology
Watson, P. F., & Petrie, A. (2010) · Theriogenology · doi:10.1016/j.theriogenology.2010.01.003 · Cited by 881
Interrater agreement and interrater reliability: Key concepts, approaches, and applications
Gisev, N., Bell, J. S., & Chen, T. F. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.04.004 · Cited by 924
A Factor Analytic Study of the Boredom Proneness Scale
Vodanovich, S. J., & Kass, S. J. (1990) · Journal of Personality Assessment · doi:10.1080/00223891.1990.9674051 · Cited by 953
The relationship between materialism and personal well-being: A meta-analysis
Dittmar, H., Bond, R., Hurst, M., & Kasser, T. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0037409 · Cited by 890
“…measures, we coded both overarching categories and main categories and then calculated interrater reliability coefficients, weighted for severity of disagreement (J. Cohen, 1968).8 The central findings for materialism were that raters obtained perfect agreement when coding measures, so that kw !”
Lauria, G., Hsieh, S. T., Johansson, O., Kennedy, W. R., Leger, J. M., Mellgren, S. I., Nolano, M., Merkies, I. S. J., Polydefkis, M., Smith, A. G., et al. (2010) · European Journal of Neurology · doi:10.1111/j.1468-1331.2010.03023.x · Cited by 864
Brugha, T. S., & Cragg, D. (1990) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.1990.tb01360.x · Cited by 883
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 852
“Computational formulas and examples have been provided by Cohen (1968), Everitt (1968), and Fleiss et al. (1969).”
BackgroundMethodSequential Analysis and Observational Methods for the Behavioral Sciences
Bakeman, R., & Quera, V. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9781139017343 · Cited by 841
Quality-of-life measures in chronic diseases of childhood
Eiser, C., & Morse, R. (2001) · Health Technology Assessment · doi:10.3310/hta5040 · Cited by 872
A Factor Analytic Study of the Boredom Proneness Scale
Vodanovich, S. J., & Kass, S. J. (1990) · Journal of Personality Assessment · doi:10.1207/s15327752jpa5501 · Cited by 871
NORMALIZING TWITTER: Journalism practice in an emerging communication space
Lasorsa, D. L., Lewis, S. C., & Holton, A. E. (2012) · Journalism Studies · doi:10.1080/1461670x.2011.571825 · Cited by 743
ROLLNICK, S., HEATHER, N., GOLD, R., & HALL, W. (1992) · British Journal of Addiction · doi:10.1111/j.1360-0443.1992.tb02720.x · Cited by 818
A standardized psychiatric interview for use in community surveys.
Goldberg, D. P., Cooper, B., Eastwood, M. R., Kedward, H. B., & Shepherd, M. (1970) · Journal of Epidemiology & Community Health · doi:10.1136/jech.24.1.18 · Cited by 805
Assessment of the adequacy of mathematical models
Tedeschi, L. O. (2006) · Agricultural Systems · doi:10.1016/j.agsy.2005.11.004 · Cited by 754
“The CCC statistic is suitable for continuous variables whereas the kappa statistics is appropriate for discrete variables (Cohen, 1960, 1968).”
MethodMood and the mundane: Relations between daily life events and self-reported mood.
Clark, L. A., & Watson, D. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.2.296 · Cited by 784
Is It Good to Cooperate? Testing the Theory of Morality-as-Cooperation in 60 Societies
Curry, O. S., Mullins, D. A., & Whitehouse, H. (2019) · Current Anthropology · doi:10.1086/701478 · Cited by 734
“005) ( Cohen 1960, 1968; Landis and Koch 1977).”
CONSIDERATIONS IN THE CHOICE OF INTEROBSERVER RELIABILITY ESTIMATES
Hartmann, D. P. (1977) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1977.10-103 · Cited by 762
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
A novel, open access method to assess sleep duration using a wrist-worn accelerometer
van Hees, V. T., Sabia, S., Anderson, K. N., Denton, S. J., Oliver, J., Catt, M., Abell, J. G., Kivimäki, M., Trenell, M. I., & Singh-Manoux, A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142533 · Cited by 744
“The agreement between weekly average accelerometer assessed sleep duration (time in bed, and total sleep duration − rounded to the nearest integer) and self-reported sleep duration from the health survey questionnaire was assessed using a confusion matrix and a weighted Cohen’s Kappa coefficient [27].”
Methodscmap: projection of single-cell RNA-seq data across data sets
(2018) · doi:10.1038/nmeth.4644 · Cited by 735
Are more resources always better for growth? Resource stickiness in market and product expansion
Mishina, Y., Pollock, T. G., & Porac, J. F. (2004) · Strategic Management Journal · doi:10.1002/smj.424 · Cited by 674
“Interrater reliability was determined using Cohen’s kappa (Cohen, 1968), which adjusts for random chance in determining interrater agreement.”
MethodLarge Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.40969 · Cited by 724
The prevalence of Prostatism: A Population-Based Survey of Urinary Symptoms
Chute, C. G., Panser, L. A., Girman, C. J., Oesterling, J. E., Guess, H. A., Jacobsen, S. J., & Lieber, M. M. (1993) · Journal of Urology · doi:10.1016/s0022-5347(17)35405-8 · Cited by 729
Domain-Specific Physical Activity and Mental Health: A Meta-analysis
White, R. L., Babic, M. J., Parker, P. D., Lubans, D. R., Astell-Burt, T., & Lonsdale, C. (2017) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.12.008 · Cited by 715
Field, T., Healy, B. T., Goldstein, S., & Guthertz, M. (1990) · Developmental Psychology · doi:10.1037/0012-1649.26.1.7 · Cited by 715
Edwards, J. R., Cable, D. M., Williamson, I. O., Lambert, L. S., & Shipp, A. J. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.4.802 · Cited by 659
“Agreement was assessed using weighted kappa (Cohen, 1968) for which disagreements of two units on the 3-point rating scale were given twice the weight of disagreements of one unit.”
MethodCopeland, J. R. M., Kelleher, M. J., Kellett, J. M., Gourlay, A. J., Gurland, B. J., Fleiss, J. L., & Sharpe, L. (1976) · Psychological Medicine · doi:10.1017/s0033291700015889 · Cited by 703
Self-efficacy and relapse in smoking cessation programs.
Condiotte, M. M., & Lichtenstein, E. (1981) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.49.5.648 · Cited by 695
Shaha, J. S., Cook, J. B., Song, D. J., Rowles, D. J., Bottoni, C. R., Shaha, S. H., & Tokish, J. M. (2015) · The American Journal of Sports Medicine · doi:10.1177/0363546515578250 · Cited by 688
“Intraobserver (ie, intrarater) reliabilities were computed by use of the Pearson correlation coefficient and were fully validated as Cohen kappa.(10,28) Analyses of variance (ie, F tests) were used for contrasting quartile means for continuous variables (eg, WOSI, SANE), followed by post hoc multiple comparison proced…”
Stirman, S. W., Miller, C. J., Toder, K., & Calloway, A. (2013) · Implementation Science · doi:10.1186/1748-5908-8-65 · Cited by 691
“The two raters overlapped on 20% of the identified segments, and we computed Cohen’s kappa coefficients [31] to determine interrater reliability for each rating category.”
MethodSystematic evaluation of rating scales for impairment and disability in Parkinson's disease
Ramaker, C., Marinus, J., Stiggelbout, A. M., & van Hilten, B. J. (2002) · Movement Disorders · doi:10.1002/mds.10248 · Cited by 666
“Where the categories are ordered, it may be preferable to give different weights to disagreements according to the magnitude of the discrepancy, the w ( weighted kappa).(16) If a squared weighting scheme is used, then the w is identical to the ICC.”
MethodMichie, S., Wood, C. E., Johnston, M., Abraham, C., Francis, J. J., & Hardeman, W. (2015) · Health Technology Assessment · doi:10.3310/hta19990 · Cited by 680
Tau-b or Not Tau-b: Measuring the Similarity of Foreign Policy Positions
Signorino, C. S., & Ritter, J. M. (1999) · International Studies Quarterly · doi:10.1111/0020-8833.00113 · Cited by 681
Neural Attentional Rating Regression with Review-level Explanations
Chen, C., Zhang, M., Liu, Y., & Ma, S. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186070 · Cited by 622
Theunissen, M., Peters, M. L., Bruce, J., Gramke, H.-F., & Marcus, M. A. (2012) · The Clinical Journal of Pain · doi:10.1097/ajp.0b013e31824549d6 · Cited by 659
“Interrater agreement on the total score was assessed using the weighted quadratic k 33 ; interrater agreement per item was assessed using the k statistic.”
The Situational Eight DIAMONDS: A taxonomy of major dimensions of situation characteristics.
Rauthmann, J. F., Gallardo-Pujol, D., Guillaume, E. M., Todd, E., Nave, C. S., Sherman, R. A., Ziegler, M., Jones, A. B., & Funder, D. C. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0037250 · Cited by 601
Circumpolar distribution and carbon storage of thermokarst landscapes
Olefeldt, D., Goswami, S., Grosse, G., Hayes, D., Hugelius, G., Kuhry, P., McGuire, A. D., Romanovsky, V. E., Sannel, A. B. K., Schuur, E. A. G., et al. (2016) · Nature Communications · doi:10.1038/ncomms13043 · Cited by 590
Bedossa, P. (2014) · Hepatology · doi:10.1002/hep.27173 · Cited by 625
“The evaluation of concordance between pathologists according to weighted kappa statistics, the percentage of overall agreement, and the number of biopsies with full agreement between all pathologists were calculated.(27) In a second session of slide reviewing, the same pathologists were given a briefing describing the…”
MethodTypes of stressful life event and the onset of anxiety and depressive disorders
Finlay-Jones, R., & Brown, G. W. (1981) · Psychological Medicine · doi:10.1017/s0033291700041301 · Cited by 641
Lobo, A., Saz, P., Marcos, G., Día, J.-L., de la Cámara, C., Ventura, T., Asín, F. M., Pascual, L. F., Montañés, J. Á., & Aznar, S. (1999) · PubMed · Cited by 669
Sekhon, M., Cartwright, M., & Francis, J. J. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-07577-3 · Cited by 541
“, sentences in transcripts) may have been coded into more than one of the categories [35, 36].”
BackgroundA Re-analysis of the Reliability of Psychiatric Diagnosis
Spitzer, R. L., & Fleiss, J. L. (1974) · British Journal of Psychiatry · doi:10.1192/bjp.125.4.341 · Cited by 612
Assessing reliability of a measure of self-rated health
Lundberg, O., & Manderbacka, K. (1996) · Scandinavian Journal of Social Medicine · doi:10.1177/140349489602400314 · Cited by 615
The Abbreviated Injury Scale and Injury Severity Score: Levels of Inter- and Intrarater Reliability
MacKenzie, E. J., Shapiro, S., & Eastham, J. N. (1985) · Medical Care · doi:10.1097/00005650-198506000-00008 · Cited by 616
Metadata
- Authors
- Cohen, J.
- Affiliations
- New York University (US)
- Corresponding author
- Cohen, Jacob
- Venue
- Psychological Bulletin
- Published
- 1968
- Volume
- 70
- Issue
- 4
- Pages
- 213-220
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2037789405
- Cited by
- 8,736
- References
- 10
- Citations by year
- 2026: 542 · 2025: 374 · 2024: 398 · 2023: 344 · 2022: 390 · 2021: 485 · 2020: 508 · 2019: 405 · 2018: 455 · 2017: 363 · 2016: 381 · 2015: 376 · 2014: 362 · 2013: 266 · 2012: 284
- FWCI
- 7.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Multidisciplinary Science and Engineering Research
- Keywords
- weighted kappa, interobserver agreement, nominal scale, partial credit, categorical data, contingency tables, inter-rater reliability
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Research Design, Models, Statistical, Likelihood Functions, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.