Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 101–200 of 1,118
Clinical Interpretation and Implications of Whole-Genome Sequencing
Dewey, F. E., Grove, M. E., Pan, C., Goldstein, B. A., Bernstein, J. A., Chaib, H., Merker, J. D., Goldfeder, R. L., Enns, G. M., David, S. P., et al. (2014) · JAMA · doi:10.1001/jama.2014.1717 · Cited by 448
The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies
Double, K. S., McGrane, J. A., & Hopfenbeck, T. N. (2020) · Educational Psychology Review · doi:10.1007/s10648-019-09510-3 · Cited by 446
Li, X., Zhang, S., Zhang, Q., Wei, X., Pan, Y., Zhao, J., Xin, X., Qin, C., Wang, X., Li, J., et al. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(18)30762-9 · Cited by 433
Hate Speech Dataset from a White Supremacy Forum
de Gibert, O., Perez, N., García-Pablos, A., & Cuadros, M. (2018) · Proceedings of the 2nd Workshop on Abusive Language Online (ALW2) · doi:10.18653/v1/w18-5102 · Cited by 425
“15 (avg k), and Fleiss’ kappa coefficient (Fleiss, 1971) (fleiss).”
Dynamics of the evolution of the strategy concept 1962–2008: a co‐word analysis
Ronda‐Pupo, G. A., & Guerras‐Martin, L. Á. (2012) · Strategic Management Journal · doi:10.1002/smj.948 · Cited by 427
“It was originally proposed by Cohen (1960) to determine the degree of agreement in the case of two raters or two methods and was generalized for the case of more than two raters by Fleiss (1971).”
MethodSpotting opinion spammers using behavioral footprints
Mukherjee, A., Kumar, A., Liu, B., Wang, J., Hsu, M., Castellanos, M., & Ghosh, R. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487580 · Cited by 426
“Additionally, we report the agreement of judges using Fleiss multi-rater kappa [11] for each method in the last row of Table 3.”
MethodEvent labeling combining ensemble detectors and background knowledge
Fanaee-T, H., & Gama, J. (2014) · Progress in Artificial Intelligence · doi:10.1007/s13748-013-0040-3 · Cited by 415
“Instead we use Fleiss’s kappa [18] which is a statistics that measures the level of agreement between multiple raters when assigning categorical ratings to a number of items or classifying items.”
MethodMurtagh, F. E., Ramsenthaler, C., Firth, A., Groeneveld, E. I., Lovell, N., Simon, S. T., Denzel, J., Guo, P., Bernhardt, F., Schildmann, E., et al. (2019) · Palliative Medicine · doi:10.1177/0269216319854264 · Cited by 413
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410
Exposing Digital Image Forgeries by Illumination Color Classification
de Carvalho, T. J., Riess, C., Angelopoulou, E., Pedrini, H., & de Rezende Rocha, A. (2013) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2013.2265677 · Cited by 406
“The kappa-value, which measures the degree of agreement between an arbitrary number of raters in deciding the class of a sample, based on the Fleiss [39] model, is 0.11.”
MethodInfluentialCriteria For Evaluating Usability Evaluation Methods
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_03 · Cited by 406
Interrater Reliability and Agreement
Tinsley, H. E. A., & Weiss, D. J. (2000) · Handbook of Applied Multivariate Statistics and Mathematical Modeling · doi:10.1016/b978-012691360-6/50005-7 · Cited by 394
Development, Use, and Psychometric Properties of the Trauma History Questionnaire
Hooper, L. M., Stockton, P., Krupnick, J. L., & Green, B. L. (2011) · Journal of Loss and Trauma · doi:10.1080/15325024.2011.572035 · Cited by 399
“According to Fleiss's ( 1971 ) guidelines, kappas in the range of .”
MethodLarge sample variance of kappa in the case of different sets of raters.
Fleiss, J. L., Nee, J. C., & Landis, J. R. (1979) · Psychological Bulletin · doi:10.1037//0033-2909.86.5.974 · Cited by 398
Self-citationParsing Argumentation Structures in Persuasive Essays
Stab, C., & Gurevych, I. (2017) · Computational Linguistics · doi:10.1162/coli_a_00295 · Cited by 388
Relation between perceived vulnerability to HIV and precautionary sexual behavior.
Gerrard, M., Gibbons, F. X., & Bushman, B. J. (1996) · Psychological Bulletin · doi:10.1037//0033-2909.119.3.390 · Cited by 396
Kriege, N. M., Johansson, F. D., & Morris, C. (2020) · Applied Network Science · doi:10.1007/s41109-019-0195-3 · Cited by 381
Employee silence motives: Investigation of dimensionality and development of measures
Brinsfield, C. T. (2013) · Journal of Organizational Behavior · doi:10.1002/job.1829 · Cited by 385
“I then calculated Fleiss’ kappa to determine agreement among the three judges (Fleiss, 1971).”
MethodVirtual Reality: A Survival Guide for the Social Scientist
Fox, J., Arena, D., & Bailenson, J. N. (2009) · Journal of Media Psychology · doi:10.1027/1864-1105.21.3.95 · Cited by 388
Fordis, M., King, J. E., Ballantyne, C. M., Jones, P. H., Schneider, K. H., Spann, S. J., Greenberg, S. B., & Greisinger, A. J. (2005) · JAMA · doi:10.1001/jama.294.9.1043 · Cited by 387
Deep Learning for the Radiographic Detection of Apical Lesions
Ekert, T., Krois, J., Meinhold, L., Elhennawy, K., Emara, R., Golla, T., & Schwendicke, F. (2019) · Journal of Endodontics · doi:10.1016/j.joen.2019.03.016 · Cited by 384
Automatic detection of cyberbullying in social media text
Van Hee, C., Jacobs, G., Emmery, C., Desmet, B., Lefever, E., Verhoeven, B., De Pauw, G., Daelemans, W., & Hoste, V. (2018) · PLOS ONE · doi:10.1371/journal.pone.0203794 · Cited by 385
“Kappa scores for the identification of cyberbullying are κ = 0.69 (Dutch) and κ = 0.59 (English).”
MethodInfluentialEffect of Outcome on Physician Judgments of Appropriateness of Care
Caplan, R. A. (1991) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1991.03460150061024 · Cited by 382
Reliability of the National Institutes of Health Stroke Scale
McLoughlin, A., Olive, P., & Lightbody, C. E. (2022) · British Journal of Neuroscience Nursing · doi:10.12968/bjnn.2022.18.sup5.s3 · Cited by 383
Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0200-9 · Cited by 377
“From the available kappa and kappa-like coefficients we chose Fleiss’ K [10] for this study because of its high flexibility.”
MethodReliability of DSM-III Anxiety Disorder Categories Using a New Structured Interview
Di Nardo, P. A. (1983) · Archives of General Psychiatry · doi:10.1001/archpsyc.1983.01790090032005 · Cited by 377
Macroscopic and histopathologic analysis of human knee menisci in aging and osteoarthritis
Pauli, C., Grogan, S. P., Patil, S., Otsuki, S., Hasegawa, A., Koziol, J., Lotz, M. K., & D’Lima, D. D. (2011) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2011.05.008 · Cited by 371
The Effect of Stroke and Stroke Prophylaxis With Aspirin or Warfarin on Quality of Life
Gage, B. F. (1996) · Archives of Internal Medicine · doi:10.1001/archinte.156.16.1829 · Cited by 370
MRI evidence of past cerebral microbleeds in a healthy elderly population
Roob, G., Schmidt, R., Kapeller, P., Lechner, A., Hartung, H.-P., & Fazekas, F. (1999) · Neurology · doi:10.1212/wnl.52.5.991 · Cited by 370
Reliability of a Complication Classification System for Orthopaedic Surgery
Sink, E. L., Leunig, M., Zaltz, I., Gilbert, J. C., & Clohisy, J. (2012) · Clinical Orthopaedics & Related Research · doi:10.1007/s11999-012-2343-2 · Cited by 362
Balkenhol, N., Waits, L. P., & Dezzani, R. J. (2009) · Ecography · doi:10.1111/j.1600-0587.2009.05807.x · Cited by 363
“To assess the agreement among methods, we used Fleiss’ Kappa statistic (Fleiss 1971).”
MethodRoute Infrastructure and the Risk of Injuries to Bicyclists: A Case-Crossover Study
Teschke, K., Harris, M. A., Reynolds, C. C. O., Winters, M., Babul, S., Chipman, M., Cusimano, M. D., Brubacher, J. R., Hunte, G., Friedman, S. M., et al. (2012) · American Journal of Public Health · doi:10.2105/ajph.2012.300762 · Cited by 358
Measuring Agreement for Multinomial Data
Davies, M., & Fleiss, J. L. (1982) · Biometrics · doi:10.2307/2529886 · Cited by 353
Self-citationNeural Question Generation from Text: A Preliminary Study
Zhou, Q., Yang, N., Wei, F., Tan, C., Bao, H., & Zhou, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-73618-1_56 · Cited by 346
Planning and financial performance of small, mature firms
Bracker, J. Y. S., & Pearson, J. N. (1986) · Strategic Management Journal · doi:10.1002/smj.4250070603 · Cited by 350
Mitropoulos, D., Artibani, W., Biyani, C. S., Bjerggaard Jensen, J., Rouprêt, M., & Truss, M. (2018) · European Urology Focus · doi:10.1016/j.euf.2017.02.014 · Cited by 349
“Survey data collected were used to calculate agreement rates and estimate the overall inter-rater agreement on all cases using Fleiss’ kappa (k) [13,14].”
MethodExploiting Burstiness in Reviews for Review Spammer Detection
Fei, G., Mukherjee, A., Liu, B., Hsu, M., Castellanos, M., & Ghosh, R. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v7i1.14400 · Cited by 348
“Additionally, we report the agreement of judges using Fleiss multi-rater kappa (Fleiss, 1971) for each method in the last row of Table 5.”
MethodPlonsky (2013) · doi:10.1017/s0272263113000399 · Cited by 347
“Cohen’s kappa is known to be overly conservative, especially when estimating reliability among a large number of categories (e.g., Aguinis, Pierce, Bosco, & Muslin, 2009 ; Brutus, Gill, & Duniewicz, 2010 ; Fleiss, 1971 ; Landis & Koch, 1977 ).”
BackgroundMiniati, M., Pistolesi, M., Marini, C., Di Ricco, G., Formichi, B., Prediletto, R., Allescia, G., Tonelli, L., Sostman, H. D., & Giuntini, C. (1996) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.154.5.8912753 · Cited by 348
A meta-analytic review of social, self-concept, and behavioral outcomes of peer-assisted learning.
Ginsburg-Block, M. D., Rohrbeck, C. A., & Fantuzzo, J. W. (2006) · Journal of Educational Psychology · doi:10.1037/0022-0663.98.4.732 · Cited by 346
“Kappas and agreement percentages were above acceptable values (Fleiss, 1971; Sattler, 2001).”
BackgroundMethodMRI white matter hyperintensities: Three-year follow-up of the Austrian Stroke Prevention Study
Schmidt, R., Fazekas, F., Kapeller, P., Schmidt, H., & Hartung, H.-P. (1999) · Neurology · doi:10.1212/wnl.53.1.132 · Cited by 342
SILVERMAN, W. K., & EISEN, A. R. (1992) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199201000-00018 · Cited by 341
Misleading Health-Related Information Promoted Through Video-Based Social Media: Anorexia on YouTube
Syed-Abdul, S., Fernandez-Luque, L., Jian, W.-S., Li, Y.-C., Crain, S., Hsu, M.-H., Wang, Y.-C., Khandregzen, D., Chuluunbaatar, E., Nguyen, P. A., et al. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2237 · Cited by 340
“Greater popularity, in terms of likes and comments, will increase the visibility of the video because search and recommender algorithms within YouTube promote highly linked and commented videos [40].”
Instruments Measuring Spirituality in Clinical Research: A Systematic Review
Monod, S., Brennan, M., Rochat, E., Martin, E., Rochat, S., & Büla, C. J. (2011) · Journal of General Internal Medicine · doi:10.1007/s11606-011-1769-7 · Cited by 338
Hermes, S., Riasanow, T., Clemons, E. K., Böhm, M., & Krcmar, H. (2020) · Business Research · doi:10.1007/s40685-020-00125-x · Cited by 330
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2734091 · Cited by 326
Radiology's Achilles' heel: error and variation in the interpretation of the Röntgen image.
Robinson, P. J. (1997) · The British Journal of Radiology · doi:10.1259/bjr.70.839.9536897 · Cited by 320
Interobserver agreement issues in radiology
Benchoufi, M., Matzner-Lober, E., Molinari, N., Jannot, A.-S., & Soyer, P. (2020) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2020.09.001 · Cited by 317
“When more than two readers are available, the Fleiss’s kappa coefficient should be used [5].”
MethodWhere in the World Are You? Geolocation and Language Identification in Twitter
Graham, M., Hale, S. A., & Gaffney, D. (2014) · The Professional Geographer · doi:10.1080/00330124.2014.907699 · Cited by 319
The COUGHVID crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms
Orlandic, L., Teijeiro, T., & Atienza, D. (2021) · Scientific Data · doi:10.1038/s41597-021-00937-4 · Cited by 312
Survey on evaluation methods for dialogue systems
Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-020-09866-x · Cited by 215
“The results show that the annotator agreement among humans was low, with a Fleiss (Fleiss 1971) = 0.3 , which shows that the task is difficult.”
MethodInfluentialLexical association measures and collocation extraction
Pecina, P. (2010) · Language Resources and Evaluation · doi:10.1007/s10579-009-9101-4 · Cited by 316
“1 An agreement measure for any numbers of annotators (Fleiss 1971): j 1⁄4 Po Pe 1 Pe ; where Po is the relative observed agreement among annotators and Pe is the theoretical probability of chance agreement (each annotator randomly choosing each category).”
BackgroundMethodQuality control for retinal OCT in multiple sclerosis: validation of the OSCAR-IB criteria
Schippling, S., Balk, L., Costello, F., Albrecht, P., Balcer, L., Calabresi, P., Frederiksen, J., Frohman, E., Green, A., Klistorner, A., et al. (2015) · Multiple Sclerosis Journal · doi:10.1177/1352458514538110 · Cited by 309
Goldstein, L. B., Jones, M. R., Matchar, D. B., Edwards, L. J., Hoff, J., Chilukuri, V., Armstrong, S. B., & Horner, R. D. (2001) · Stroke · doi:10.1161/01.str.32.5.1091 · Cited by 308
Systematic review and meta-analysis methodology
Crowther, M., Lim, W., & Crowther, M. A. (2010) · Blood · doi:10.1182/blood-2010-05-280883 · Cited by 306
LYNEHAM, H. J., ABBOTT, M. J., & RAPEE, R. M. (2007) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/chi.0b013e3180465a09 · Cited by 307
“statistic for multiple raters (Fleiss, 1971) with the statistic program STATA 9.”
MethodLIRIS-ACCEDE: A Video Database for Affective Content Analysis
Baveye, Y., Dellandrea, E., Chamaret, C., & Chen, L. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2396531 · Cited by 306
“Several measures of inter-annotator agreement are used in the literature such as percent agreement, Fleiss’ kappa [46] and Krippendorff’s alpha [47].”
Eguchi, T., Kameda, K., Lu, S., Bott, M. J., Tan, K. S., Montecalvo, J., Chang, J. C., Rekhtman, N., Jones, D. R., Travis, W. D., et al. (2019) · Journal of Thoracic Oncology · doi:10.1016/j.jtho.2018.09.005 · Cited by 300
“030 2000–2004 244 (25) 63 (12) 45 (13) 46 (13) 2005–2009 341 (35) 179 (34) 117 (34) 121 (35) 2010–2014 317 (33) 263 (50) 172 (49) 167 (48) Age at surgery, y 69 (61–75) 70 (64–76) 0.”
Kapeller, P., Barber, R., Vermeulen, R. J., Adèr, H., Scheltens, P., Freidl, W., Almkvist, O., Moretti, M., del Ser, T., Vaghfeldt, P., et al. (2003) · Stroke · doi:10.1161/01.str.0000049766.26453.e9 · Cited by 300
(Comet-) Atomic 2020: On Symbolic and Neural Commonsense Knowledge Graphs
Hwang, J. D., Bhagavatula, C., Le Bras, R., Da, J., Sakaguchi, K., Bosselut, A., & Choi, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16792 · Cited by 286
Sales and operations planning: A research synthesis
Tavares Thomé, A. M., Scavarda, L. F., Fernandez, N. S., & Scavarda, A. J. (2012) · International Journal of Production Economics · doi:10.1016/j.ijpe.2011.11.027 · Cited by 297
Sheridan, S. M., Smith, T. E., Moorman Kim, E., Beretvas, S. N., & Park, S. (2019) · Review of Educational Research · doi:10.3102/0034654318825437 · Cited by 296
Zaidan, O. F., & Callison-Burch, C. (2014) · Computational Linguistics · doi:10.1162/coli_a_00169 · Cited by 294
Boundary object or bridging concept? A citation network analysis of resilience
Baggio, J. A., Brown, K., & Hellebrandt, D. (2015) · Ecology and Society · doi:10.5751/es-07484-200202 · Cited by 297
Understanding FinTech start-ups – a taxonomy of consumer-oriented service offerings
Gimpel, H., Rau, D., & Röglinger, M. (2018) · Electronic Markets · doi:10.1007/s12525-017-0275-0 · Cited by 294
Behavioral Signal Processing: Deriving Human Behavioral Informatics From Speech and Language
Narayanan, S., & Georgiou, P. G. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2236291 · Cited by 294
“Interannotator reliability can be measured in many different ways, e.g., using percentage agreement, Holsti's method [25], Scott's [26], Cohen's [27], Fleiss' f [28], and Krippendorff's [29].”
MethodJames, S. (2004) · Social Service Review · doi:10.1086/424546 · Cited by 292
SemEval-2018 Task 3: Irony Detection in English Tweets
Van Hee, C., Lefever, E., & Hoste, V. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1005 · Cited by 292
“As metric, we used Fleiss’ Kappa (Fleiss, 1971), a widespread statistical measure in the field of computational linguistics for assessing annotator agreement on categorical ratings (Carletta, 1996).”
MethodEvaluation and Measurement of Software Process Improvement—A Systematic Literature Review
Unterkalmsteiner, M., Gorschek, T., Islam, A. K. M. M., Cheng, C. K., Permadi, R. B., & Feldt, R. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.26 · Cited by 291
“Since the focus of this review is the measurement and evaluation of SPI (see our research questions in Table 1), general discussions on improvement approaches and comparisons of frameworks or models were excluded.”
BackgroundJohnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289
“Interrater agreement was evaluated by Fleiss Œ (5).”
MethodPsychedelic therapy for smoking cessation: qualitative analysis of participant accounts
Noorani, T., Garcia-Romeu, A., Swift, T. C., Griffiths, R. R., & Johnson, M. W. (2018) · Journal of Psychopharmacology · doi:10.1177/0269881118780612 · Cited by 285
Banerjee, I., Ling, Y., Chen, M. C., Hasan, S. A., Langlotz, C. P., Moradzadeh, N., Chapman, B., Amrhein, T., Mong, D., Rubin, D. L., et al. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2018.11.004 · Cited by 285
“Inter-rater reliability among the three raters for annotation of Stanford, Duke and Colorado datasets was calculated using Fleiss’ kappa [31] – a generalization of Cohen's Kappa for more than two raters.”
MethodMeta-analysis of Cohen’s kappa
Sun, S. (2011) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-011-0077-3 · Cited by 282
“…been extended to estimate inter-rater reliability for many complicated rating scenarios (e.g., Berry and Mielke 1998; Cohen 1968, 1972; Davies and Fleiss 1982; Fleiss 1971; Gross 1986; Janson and Olsson 2001, 2004; Kraemer 1980; Kraemer et al. 2004; Landis and Koch 1977b; Vanbelle and Albert 2009).”
BackgroundBalint, P. V., Terslev, L., Aegerter, P., Bruyn, G. A. W., Chary-Valckenaere, I., Gandjbakhch, F., Iagnocco, A., Jousse-Joulin, S., Möller, I., Naredo, E., et al. (2018) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2018-213609 · Cited by 279
MEADE, M. O., COOK, R. J., GUYATT, G. H., GROLL, R., KACHURA, J. R., BEDARD, M., COOK, D. J., SLUTSKY, A. S., & STEWART, T. E. (2000) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.161.1.9809003 · Cited by 280
“Because of this problem with raw agreement, we calculated chancecorrected agreement, using the k statistic ( 13 ).”
MethodMarzi, G., Balzano, M., & Marchiori, D. (2024) · MethodsX · doi:10.1016/j.mex.2023.102545 · Cited by 292
The Effects of Simplified and Elaborated Texts on Foreign Language Reading Comprehension
Yano, Y., Long, M. H., & Ross, S. (1994) · Language Learning · doi:10.1111/j.1467-1770.1994.tb01100.x · Cited by 279
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.234 · Cited by 279
Marcus, A., Wu, E., Karger, D., Madden, S., & Miller, R. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/2047485.2047487 · Cited by 278
“As a quantitative measure of IRR, we utilize Fleiss’ κ [4].”
MethodIncreasing decision relevance of ecosystem service science
Mandle, L., Shields-Estrada, A., Chaplin-Kramer, R., Mitchell, M. G. E., Bremer, L. L., Gourevitch, J. D., Hawthorne, P., Johnson, J. A., Robinson, B. E., Smith, J. R., et al. (2020) · Nature Sustainability · doi:10.1038/s41893-020-00625-y · Cited by 274
The need to separate the wheat from the chaff in medical informatics
Cabitza, F., & Campagner, A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104510 · Cited by 276
RAPEE, R. M., BARRETT, P. M., DADDS, M. R., & EVANS, L. (1994) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199409000-00008 · Cited by 273
Content Analysis—A Methodological Primer for Gender Research
Neuendorf, K. A. (2011) · Sex Roles · doi:10.1007/s11199-010-9893-0 · Cited by 272
“Cohen’s kappa (or multi-coder kappa; Cohen 1960; Fleiss 1971) is a widely used coefficient for nominal/categorical data (Lombard et al.”
MethodCorona, R., Mele, A., Amini, M., De Rosa, G., Coppola, G., Piccardi, P., Fucci, M., Pasquini, P., & Faraggiana, T. (1996) · Journal of Clinical Oncology · doi:10.1200/jco.1996.14.4.1218 · Cited by 266
Donner, A., & Eliasziw, M. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111109 · Cited by 266
Lamb, M. E., Thompson, R. A., Gardner, W. P., Charnov, E. L., & Estes, D. (1984) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00026522 · Cited by 266
When guests trust hosts for their words: Host description and trust in sharing economy
Tussyadiah, I. P., & Park, S. (2018) · Tourism Management · doi:10.1016/j.tourman.2018.02.002 · Cited by 264
“, inter-rater reliability), Fleiss’ Kappa value (Fleiss, 1971) was calculated for each destination.”
MethodTowards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
“raters [183] can be used to assess agreement beyond chance between these raters and serve as an indication for the reliability of the ratings.”
BackgroundKleeck, A. V., Gillam, R. B., Hamilton, L., & McGrath, C. (1997) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jslhr.4006.1261 · Cited by 264
Impact of the usage of social media in the workplace on team and employee performance
Song, Q., Wang, Y., Chen, Y., Benitez, J., & Hu, J. (2019) · Information & Management · doi:10.1016/j.im.2019.04.003 · Cited by 260
Estimating Intraclass Correlation for Binary Data
Ridout, M. S., Demétrio, C. G. B., & Firth, D. (1999) · Biometrics · doi:10.1111/j.0006-341x.1999.00137.x · Cited by 262
“Asymptotic equivalence of fi~ov and f i ~ ~ is demonstrated by Fleiss and Cuzick (1979), and it can be shown that ~ U B is also asymptotically equivalent t o these two estimators.”
BackgroundNew State of Play in Information Systems Research: The Push to the Edges1
Grover, V., & Lyytinen, K. (2015) · MIS Quarterly · doi:10.25300/misq/2015/39.2.01 · Cited by 260
MasakhaNER: Named Entity Recognition for African Languages
Adelani, D. I., Abbott, J., Neubig, G., D’souza, D., Kreutzer, J., Lignos, C., Palen-Michel, C., Buzaaba, H., Rijhwani, S., Ruder, S., et al. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00416 · Cited by 257
Dependence of Weighted Kappa Coefficients on the Number of Categories
Brenner, H., & Kliebsch, U. (1996) · Epidemiology · doi:10.1097/00001648-199603000-00016 · Cited by 255
Zhang, S., Faries, D. E., Vowles, M., & Michelson, D. (2005) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.7 · Cited by 255
Development of the clinical assessment scale in autoimmune encephalitis
Lim, J., Lee, S., Moon, J., Jun, J., Kim, T., Shin, Y., Abdullah, S., Byun, J., Sunwoo, J., Kim, K. T., et al. (2019) · Annals of Neurology · doi:10.1002/ana.25421 · Cited by 253
Mak, A., Cheung, M. W.-L., Cheak, A. A.-C., & Ho, R. C.-M. (2010) · Rheumatology · doi:10.1093/rheumatology/kep373 · Cited by 255
Cranial Nonmetric Variation and Estimating Ancestry*
Hefner, J. T. (2009) · Journal of Forensic Sciences · doi:10.1111/j.1556-4029.2009.01118.x · Cited by 252
“Fleiss’ kappa (29) works for any number of observers and is appropriate”
BackgroundToTTo: A Controlled Table-To-Text Generation Dataset
Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.89 · Cited by 252
How do medical residents discuss resuscitation with patients?
Tulsky, J. A., Chesney, M. A., & Lo, B. (1995) · Journal of General Internal Medicine · doi:10.1007/bf02599915 · Cited by 252
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.