Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 801–900 of 1,118
An evaluation of three nutritional screening tools in a Portuguese oncology centre
do Amaral, T. F., Antunes, A., Cabral, S., Alves, P., & Kent‐Smith, L. (2008) · Journal of Human Nutrition and Dietetics · doi:10.1111/j.1365-277x.2008.00917.x · Cited by 112
Emotional Congruence in Sexual Offenders Against Children
Wilson, R. J. (1999) · Sexual Abuse · doi:10.1177/107906329901100104 · Cited by 86
Large Measles Outbreak at a German Public School, 2006
Wichmann, O., Hellenbrand, W., Sagebiel, D., Santibanez, S., Ahlemeyer, G., Vogt, G., Siedler, A., & van Treeck, U. (2007) · The Pediatric Infectious Disease Journal · doi:10.1097/inf.0b013e318060aca1 · Cited by 86
“The overall agreement of vaccination status as ascertained from the vaccination records versus the questionnaire was measured by calculating Fleiss’ Kappa coefficient with values ranging from 0 to 1, with 1 indicating perfect agreement and 0 indicating no agreement beyond that expected by chance.(9)”
MethodEvaluation of the clinical value of pharmacists’ modifications of prescription errors
Buurma, H., Smet, P. A. G. M. D., Leufkens, H. G. M., & Egberts, T. C. G. (2004) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2004.02181.x · Cited by 102
Schofield, E. (2003) · Brain · doi:10.1093/brain/awg085 · Cited by 86
Stevens, J. P., Kachniarz, B., Wright, S. B., Gillis, J., Talmor, D. S., Clardy, P. F., & Howell, M. D. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e3182a66903 · Cited by 92
“We compared interobserver variation among participants using Fleiss κ statistic (14).”
MethodConstructing Corpora for the Development and Evaluation of Paraphrase Systems
Cohn, T., Callison-Burch, C., & Lapata, M. (2008) · Computational Linguistics · doi:10.1162/coli.08-003-r1-07-044 · Cited by 103
“Either a separate distribution is estimated for each coder (Cohen 1960) or the same distribution for all coders (Scott 1955; Fleiss 1971; Siegel and Castellan 1988).”
BackgroundA Hybrid Retrieval-Generation Neural Conversation Model
Yang, L., Hu, J., Qiu, M., Qu, C., Gao, J., Croft, W. B., Liu, X., Shen, Y., & Liu, J. (2019) · Proceedings of the 28th ACM International Conference on Information and Knowledge Management · doi:10.1145/3357384.3357881 · Cited by 81
“is evaluated by Fleiss’ kappa [5] which is a statistical measure of inter-rater consistency.”
MethodAssessing Sentiment of Text by Semantic Dependency and Contextual Valence Analysis
Shaikh, M. A. M., Prendinger, H., & Mitsuru, I. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_18 · Cited by 100
“Here, c(w) = Prior Valence of word w, whereby -5 ≤ c(w) ≤ +5 m = Number of judges (8) pi = The number of positive senses assigned by i-th judge, for word w ni = The number of negative senses assigned by i-th judge, for word w Ni = Total number of senses counted by i-th judge for word w To measure inter-agreement among…”
MethodThe accuracy of pre‐tender building cost estimates in Australia
Aibinu, A. A., & Pasco, T. (2008) · Construction Management and Economics · doi:10.1080/01446190802527514 · Cited by 88
“It is a measure of the degree of agreement that can be expected above chance (Fleiss, 1971).”
BackgroundMethodChildren’s knowledge of the earth: A new methodological and statistical approach
Straatemeier, M., van der Maas, H. L. J., & Jansen, B. R. J. (2008) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2008.03.004 · Cited by 88
“Fleiss’s kappa (Fleiss, 1971) for the model classifications was .76, p .05.”
MethodTerslev, L., Gutiérrez, M., Christensen, R., Balint, P. V., Bruyn, G. A. W., Sedie, A. D., Filippucci, E., Garrido, J., Hammer, H. B., Iagnocco, A., et al. (2015) · The Journal of Rheumatology · doi:10.3899/jrheum.150366 · Cited by 93
Tageldin, A., Sayed, T., & Wang, X. (2015) · Transportation Research Record Journal of the Transportation Research Board · doi:10.3141/2520-19 · Cited by 87
“Under the hypothesis of no agreement beyond chance and using the central limit theorem, the ( ) κ κ / var may be approximately distributed as a standard normal variant (39).”
MethodGwet, K. L. (2008) · Wiley eBooks · doi:10.1002/9780471462422.eoct631 · Cited by 93
Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus
Soleymani, M., & Larson, M. (2010) · Archive ouverte UNIGE (University of Geneva) · Cited by 100
Enhancing reading skills through a video game mixing action mechanics and cognitive training
Pasqualotto, A., Altarelli, I., Angeli, A. D., Menestrina, Z., Bavelier, D., & Venuti, P. (2022) · Nature Human Behaviour · doi:10.1038/s41562-021-01254-x · Cited by 107
“Fleiss’ kappa procedure 73 was used to determine reliability.”
MethodDecision development in facilitated modelling workshops
Franco, L. A., & Rouwette, E. A. J. A. (2011) · European Journal of Operational Research · doi:10.1016/j.ejor.2011.01.039 · Cited by 84
“Extensions of Cohen’s kappa for more than two coders can be found in Fleiss (1971).”
BackgroundTursi, A., Brandimarte, G., Mario, F. D., Andreoli, A., Annunziata, M. L., Astegiano, M., Bianco, M. A., Buri, L., Cammarota, G., Capezzuto, E., et al. (2014) · Digestive Diseases · doi:10.1159/000366039 · Cited by 84
Measuring pairwise interobserver agreement when all subjects are judged by the same observers
Schouten, H. J. A. (1982) · Statistica Neerlandica · doi:10.1111/j.1467-9574.1982.tb00774.x · Cited by 84
Havas, C. R., Resch, B., Francalanci, C., Pernici, B., Scalia, G., Fernandez-Marquez, J. L., Achte, T. V., Zeug, G., Mondardini, M., Grandoni, D., et al. (2017) · Sensors · doi:10.3390/s17122766 · Cited by 98
“, Fleiss Kappa statistics [102] can measure the correlation between the classification of different people.”
BackgroundMartin, R., Davis, G. M., Baron, R. S., & Suls, J. (1994) · Health Psychology · doi:10.1037//0278-6133.13.5.432 · Cited by 84
Substance Abuse and Psychiatric Disorders in Prison Inmates
Chiles, J. A., CLEVE, E. V., Jemelka, R. P., & Trupin, E. W. (1990) · Psychiatric Services · doi:10.1176/ps.41.10.1132 · Cited by 84
BanFakeNews: A Dataset for Detecting Fake News in BanglaPreprint
Hossain, Z., Rahman, A., Islam, M. S., & Kar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08789 · Cited by 159
“The inter-annotator agreement, measured as Fleiss’ Kappa(Fleiss, 1971) is 38.83% and mean pairwise Cohen’s Kappa(Cohen, 1960) is 39.05% which indicates that our human annotators have given the same answer on nearly 39% percent of the questions.”
MethodInfluentialIntensive behavioral group treatment of avoidant personality disorder
Renneberg, B., Goldstein, A. J., Phillips, D., & Chambless, D. L. (1990) · Behavior Therapy · doi:10.1016/s0005-7894(05)80337-5 · Cited by 83
Stapleton, C., Hough, P., Oldmeadow, L. B., Bull, K., Hill, K., & Greenwood, K. M. (2009) · Australian Journal on Ageing · doi:10.1111/j.1741-6612.2009.00375.x · Cited by 83
Using global maps to predict the risk of dengue in Europe
Rogers, D. J., Suk, J. E., & Semenza, J. C. (2013) · Acta Tropica · doi:10.1016/j.actatropica.2013.08.008 · Cited by 99
“All three maps were compared using Fleiss’s kappa which allows comparisons of more than two raters (=maps here) (Fleiss, 1971); this comparison was done both globally and by three regions; the Americas (longitudes 180W to 30W), Africa and Europe (longitudes 30W to 60E) and India and SE Asia…”
MethodNatural Sentences as Valid Units for Coded Political Texts
Däubler, T., Benoit, K. R., Mikhaylov, S., & Laver, M. (2012) · British Journal of Political Science · doi:10.1017/s0007123412000105 · Cited by 98
On Similarity Coefficients for 2×2 Tables and Correction for Chance
Warrens, M. J. (2008) · Psychometrika · doi:10.1007/s11336-008-9059-y · Cited by 84
“Multivariate versions of Cohen’s kappa (SCohen) can for instance be found in Fleiss (1971), Light (1971), Popping (1983), and Heuvelmans and Sanders (1993).”
BackgroundDiagnosing Dengue at the Point-of-Care: Utility of a Rapid Combined Diagnostic Kit in Singapore
Gan, V. C. H., Tan, L.-K., Lye, D. C., Pok, K.-Y., Mok, S.-Q., Chua, R., Leo, Y., & Ng, L. C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090037 · Cited by 83
“Case definitions for probable dengue by clinical criteria used were WHO 1997 (fever with any two of headache, retro-orbital pain, myalgia, arthralgia, rash, haemorrhagic manifestations, or leukopaenia) [21] and WHO 2009 (fever with two of nausea or vomiting; rash; aches and pains; tourniquet test positive; leucopenia;…”
MethodFrom Arguments to Key Points: Towards Automatic Argument Summarization
Bar-Haim, R., Eden, L., Friedman, R., Kantor, Y., Lahav, D., & Slonim, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.371 · Cited by 168
Smetanin, S. (2020) · IEEE Access · doi:10.1109/access.2020.3002215 · Cited by 103
“In addition to sentiment annotation guidelines, it is highly recommended to publish an inter-annotator agreement measure such as Fleiss’ kappa [181] or Krippendorff’s alpha [182] as well as other details of the annotation process.”
MethodUnderstanding the impact of trade policy effect uncertainty on firm‐level innovation investment
Chen, D. Q., Hu, N., Liang, P., & Swink, M. L. (2023) · Journal of Operations Management · doi:10.1002/joom.1285 · Cited by 98
Generating Automatic Feedback on UI Mockups with Large Language Models
Duan, P., Warner, J., Li, Y., & Hartmann, B. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642782 · Cited by 83
“To characterize subjectivity in our study, we computed inter-rater reliability using Fleiss’ Kappa [13].”
Flodén, J. (2025) · British Educational Research Journal · doi:10.1002/berj.4069 · Cited by 82
The improvement of internal consistency of the Acoustic Voice Quality Index
Latoszek, B. B. V., & Maryn, Y. (2015) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2015.04.012 · Cited by 96
“To assess the agreement of inter-rater reliability from all VDs among the five judges, we computed the kappa coefficient according to Fleiss [31], who extended the Ck for more than two judges.”
MethodLebedeva, A., Westman, E., Borza, T., Beyer, M. K., Engedal, K., Aarsland, D., Selbæk, G., & Haberg, A. K. (2017) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2017.00013 · Cited by 91
“Kappa is suggested to provide more reliable information regarding the classifier performance than actual accuracy in case of different class distribution in the dataset (Fleiss, 1971).”
BackgroundAshley, P. F., Blinkhorn, A. S., & Davies, R. M. (1998) · Journal of Dentistry · doi:10.1016/s0300-5712(97)00007-9 · Cited by 83
Low-Resource Knowledge-Grounded Dialogue Generation
Zhao, X., Wu, W., Tao, C., Xu, C., Zhao, D., & Yan, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10348 · Cited by 78
“Each response receives 3 scores on each aspect, and agreement among the annotators are calculated with Fleiss’ kappa (Fleiss, 1971).”
MethodIruskieta, M., da Cunha, I., & Taboada, M. (2014) · Language Resources and Evaluation · doi:10.1007/s10579-014-9271-6 · Cited by 106
“15 For Kappa segment candidates were calculated automatically by counting verbs.”
BackgroundInfluentialAvanzi, M., Obin, N., Lacheret, A., & Victorri, B. (2011) · Interspeech · doi:10.21437/interspeech.2011-534 · Cited by 35
“The standard kappa [19] does not give good results: the value is around 0.”
BackgroundClustering Mobile Apps Based on Mined Textual Features
Al-Subaihin, A. A., Sarro, F., Black, S., Capra, L., Harman, M., Jia, Y., & Zhang, Y. (2016) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962600 · Cited by 98
“Alter-natively, Fleiss’ Kappa [18] is used when there are more than two raters; however, it is only suited when the rating system is nominal or categorical.”
MethodManagement of Service Level Agreements for Cloud Services in IoT: A Systematic Mapping Study
Mubeen, S., Asadollah, S. A., Papadopoulos, A. V., Ashjaei, M., Pei-Breivold, H., & Behnam, M. (2017) · IEEE Access · doi:10.1109/access.2017.2744677 · Cited by 97
“Later, the limitation of tworaters was eliminated by Fleiss [362], who generalized the method for multi-raters.”
MethodInfluentialWireframe-based UI Design Search through Image Autoencoder
Chen, J., Chen, C., Xing, Z., Xia, X., Zhu, L., Grundy, J., & Wang, J. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3391613 · Cited by 123
“For the second metric, we compute Fleiss’s kappa statistics [38], which is used to evaluate the agreement between multiple raters.”
Method"How over is it?" Understanding the Incel Community on YouTube
Papadamou, K., Zannettou, S., Blackburn, J., De Cristofaro, E., Stringhini, G., & Sirivianos, M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479556 · Cited by 80
“We also compute the Fleiss’ Kappa Score [26] to assess the agreement between the annotators, finding it to be 0 .”
MethodMetaPro: A computational metaphor processing model for text pre-processing
Mao, R., Li, X., Ge, M., & Cambria, E. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.06.002 · Cited by 111
“19 We employ Fleiss [89] kappa (κ) as an greement measure for human evaluations, where κcoh.”
MethodSchopow, N., Osterhoff, G., & Baur, D. (2023) · JMIR Medical Informatics · doi:10.2196/48933 · Cited by 80
“The extracted data were compared using Fleiss and Cohen κ , correlating results from both human researchers and 2 iterations of ChatGPT 3.5 [35,36].”
MethodInfluentialPrevalence of diabetic retinopathy in children and adolescents with IDDM
Kernell, A., Dedorsson, I., Johansson, B., WICKSTRÖM, C. P., Ludvigsson, J., Tuvemo, T., Neiderud, J., Sjöström, K., Malmgren, K., Kanulf, P., et al. (1997) · Diabetologia · doi:10.1007/s001250050679 · Cited by 82
“Weighted kappa [12] for total agreement between three raters for microaneurysms and the KROC-grading system was 0.”
BackgroundDong, X., Simon, M. A., & Evans, D. A. (2009) · Gerontology · doi:10.1159/000243164 · Cited by 83
“showed that the self-neglect measure was tested using the κ statistic algorithm [”
MethodAssessment of the Scope and Quality of Clinical Practice Guidelines in Lung Cancer*
Harpole, L. H., Kelley, M. J., Schreiber, G., Toloza, E. M., Kolimaga, J. T., & McCrory, D. C. (2003) · CHEST Journal · doi:10.1378/chest.123.1_suppl.7s · Cited by 82
Landefeld, C. S., Anderson, P. A., Goodnough, L. T., MOIR, T. W., Hom, D. L., Rosenblatt, M. W., & Goldman, L. (1989) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(89)90066-8 · Cited by 82
Trans-national horizon scanning for invasive non-native species: a case study in western Europe
Gallardo, B., Zieritz, A., Adriaens, T., Bellard, C., Boets, P., Britton, J. R., Newman, J. R., Valkenburg, J. L. C. H. V., & Aldridge, D. C. (2015) · Biological Invasions · doi:10.1007/s10530-015-0986-0 · Cited by 101
Helping developers help themselves: automatic decomposition of code review changesets
Barnett, M., Bird, C., Brunet, J., & Lahiri, S. K. (2015) · doi:10.5555/2818754.2818773 · Cited by 81
Agreement in Classifying Bloodstream Infections Among Multiple Reviewers Conducting Surveillance
Mayer, J., Greene, T., Howell, J., Ying, J., Rubin, M. A., Trick, W. E., & Samore, M. (2012) · Clinical Infectious Diseases · doi:10.1093/cid/cis410 · Cited by 84
How Information Snowballs: Exploring the Role of Exposure in Online Rumor Propagation
Arif, A., Shanahan, K., Chou, F.-J., Dosouto, Y., Starbird, K., & Spiro, E. S. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2819964 · Cited by 81
“An inter-rater reliability analysis using the Fleiss’ Kappa statistic [8] was performed to ascertain”
MethodChen, Y., Xing, X., Lin, J., Zheng, H., Wang, Z., Liu, Q., & Xu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.83 · Cited by 76
Chen, Y., Li, J., Shan, S., Wang, M., & Hong, R. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3453443 · Cited by 78
“Additionally, we have conducted an inter-annotator agreement analysis on the FERPlus [17] and DFEW [4] datasets using the Fleiss’ Kappa [20] metric.”
MethodZaina, F., Négrini, S., & Atanasio, S. (2009) · Scoliosis · doi:10.1186/1748-7161-4-3 · Cited by 86
“Despite published controversies [14-16], Kappa statistics are still widely used.”
MethodAssessment of Interjudge Reliability in the Open-Ended Questions Coding Process
Muñoz‐Leiva, F., Montoro-Ríos, F. J., & Martínez, T. L. (2006) · Quality & Quantity · doi:10.1007/s11135-005-1093-6 · Cited by 103
“Tel: +34-958-249603; Fax: +34- 958-240695; E-mail: franml@ugr.es”
BackgroundZenk, S. N., Schulz, A. J., Mentz, G. B., House, J. S., Gravlee, C. C., Miranda, P. Y., Miller, P., & Kannan, S. (2006) · Health & Place · doi:10.1016/j.healthplace.2006.05.003 · Cited by 86
“The now widely accepted extension of Kappa proposed by Fleiss (1971) when the number of raters is three or more is not available in SAS version 8 (SAS Institute Inc., Cary, NC).”
BackgroundThe Relationship between Values and Appeals in Israeli Advertising: A Smallest Space Analysis
Hetsroni, A. (2000) · Journal of Advertising · doi:10.1080/00913367.2000.10673617 · Cited by 102
Statistical description of interrater variability in ordinal ratings
Nelson, J. C., & Pepe, M. S. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900505 · Cited by 81
Petrakos, M., Benediktsson, J. A., & Kanellopoulos, I. (2001) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/36.964992 · Cited by 81
“One approach [9] to overcome the complexity is to divide the problem into multiple pairwise ones and compute the average of the kappas.”
BackgroundGeneral Observer-Agreement Measures on Individual Subjects and Groups of Subjects
O'Connell, D. L., & Dobson, A. J. (1984) · Biometrics · doi:10.2307/2531148 · Cited by 81
Privee: An Architecture for Automatically Analyzing Web Privacy PoliciesPreprint
Zimmeck, S., & Bellovin, S. M. (2014) · SSRN Electronic Journal · Cited by 120
Zander, E., Willfors, C., Berggren, S., Olsson, N. C., Coco, C., Elmund, A., Moretti, Å. H., Holm, A., Jifält, I., Kosieradzki, R., et al. (2015) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-015-0793-2 · Cited by 96
Puanlayıcılar Arası Güvenirlik Belirleme Tekniklerinin Karşılaştırılması
Bilgen, Ö. B., & Doğan, N. (2017) · Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi · doi:10.21031/epod.294847 · Cited by 91
Talan, T., Doğan, Y., & Batdı, V. (2020) · Journal of Research on Technology in Education · doi:10.1080/15391523.2020.1743798 · Cited by 134
The TiltMeter app is a novel and accurate measurement tool for the weight bearing lunge test
Williams, C., Caserta, A., & Haines, T. P. (2013) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2013.02.001 · Cited by 80
Characteristics of useful code reviews: an empirical study at Microsoft
Bosu, A., Greiler, M., & Bird, C. (2015) · Mining Software Repositories · doi:10.5555/2820518.2820538 · Cited by 79
Baumann, I., Führer, M., Behrendt, S., Campr, V., Csomor, J., Furlan, I., de Haas, V., Kerndrup, G. B., Leguit, R. J., Paepe, P. D., et al. (2012) · Histopathology · doi:10.1111/j.1365-2559.2011.04156.x · Cited by 79
“Fleiss’ kappa was used as statistical measure for assessing the reliability of agreement between a fixed number of raters.(11) Interpretation of the kappa coefficient was based on guidelines provided by Landis and Koch.”
MethodAssessment of representational competence in kinematics
Klein, P., Müller, A., & Kühn, J. (2017) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.13.010132 · Cited by 87
“After each rater assigned one of the six categories (p, g, f, p ↔ f, p ↔ g, and g ↔ f) to each item, Fleiss’κF was calculated as a measure of interrater agreement [20].”
MethodShrivastava, K., Kumar, S., & Jain, D. K. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-07813-9 · Cited by 90
Dong, X., Simon, M. A., Wilson, R. S., de Leon, C. F. M., Rajan, K. B., & Evans, D. A. (2010) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2010.03156.x · Cited by 95
Cognitive profile analysis: A shared professional myth.
Watkins, M. W. (2000) · School Psychology Quarterly · doi:10.1037/h0088802 · Cited by 79
“For the two significant profile types identified by Stanton and Reynolds (2000), partial K statistics (Fleiss, 1971) were -.”
MethodGlance: rapidly coding behavioral video with the crowd
Lasecki, W. S., Gordon, M., Koutra, D., Jung, M. F., Dow, S. P., & Bigham, J. P. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647367 · Cited by 94
“• A common way to measure agreement level is to use a kappa score to measure inter-rater reliability [17].”
BackgroundSingh, S., Pinsky, P., Fineberg, N. S., Gierada, D. S., Garg, K., Sun, Y., & Nath, P. H. (2011) · Radiology · doi:10.1148/radiol.10101254 · Cited by 79
Wijburg, M. T., Lissenberg-Witte, B. I., Vennegoor, A., Roosendaal, S. D., Sánchez, E., Liu, Y., Jarnalo, C. O. M., Uitdehaag, B. M. J., Barkhof, F., Killestein, J., et al. (2016) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp-2016-313772 · Cited by 84
Cluster analysis of depressed college students' social behaviors.
Hokanson, J. E., & Butler, A. C. (1992) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.62.2.273 · Cited by 79
Pause, B. M., Sojka, B., & Ferstl, R. (1997) · Chemical Senses · doi:10.1093/chemse/22.1.9 · Cited by 80
“Changes in the verbal descriptions due to odor concentration differences were analyzed by determining the degree of agreement (k) between the raters (Fleiss, 1971), separately for each concentration step.”
MethodTwo are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems
Song, Y., Yan, R., Li, X., Zhao, D., & Zhang, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07149 · Cited by 78
Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N. G., Venugopal, V., Mahajan, V., Rao, P., & Warier, P. (2018) · arXiv.org · doi:10.48550/arxiv.1803.05854 · Cited by 132
“In addition, we measured concordance between all the three readers on each finding using Fleiss’ kappa( κ ) [50] statistic.”
MethodImproving Factual Consistency of Abstractive Summarization via Question Answering
Nan, F., Nogueira dos Santos, C., Zhu, H., Ng, P., McKeown, K., Nallapati, R., Zhang, D., Wang, Z., Arnold, A. O., & Xiang, B. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.536 · Cited by 78
“Fleiss’s Kappa (Fleiss et al., 1971) shows fair agreement for factual consistency, informativeness and grammatical correctness choices (0.136/0.270/0.043 for XSUM and 0.237/0.202/0.206 for CNNDM).”
BackgroundInfluentialSiegert, I., Böck, R., & Wendemuth, A. (2013) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-013-0129-9 · Cited by 90
“To calculate the agreement, mostly a kappa-like statistic is used, like Cohen’s kappa κ [10] for a two annotator case and Fleiss’ kappa K [24] for multiple annotators.”
MethodOn the Role of Discourse Markers for Discriminating Claims and Premises in Argumentative Discourse
Eckle‐Kohler, J., Kluge, R., & Gurevych, I. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1267 · Cited by 102
“3 The token-based kappa κt metric treats each token as annotation item and, thereon, calculates Fleiss’ kappa (Fleiss, 1971).”
MethodJansen, I. G. H., van Vuuren, A. B., van Zwam, W. H., van den Wijngaard, I. R., Berkhemer, O. A., Lingsma, H. F., Slump, C. H., van Oostenbrugge, R. J., Treurniet, K. M., Dippel, D. W. J., et al. (2017) · Radiology · doi:10.1148/radiol.2017162445 · Cited by 78
A Large Cross-Sectional Survey Study of the Prevalence of Alopecia Areata in the United States
Benigno, M., Anastassopoulos, K. P., Mostaghimi, A., Udall, M., Daniel, S. R., Cappelleri, J. C., Chander, P., Wahl, P. M., Lapthorn, J., Kauffman, L., et al. (2020) · Clinical Cosmetic and Investigational Dermatology · doi:10.2147/ccid.s245649 · Cited by 91
Twitter Emotion Analysis in Earthquake Situations.
Vo, B.-K. H., & Collier, N. (2013) · Cited by 110
Aminopyridines for symptomatic treatment in multiple sclerosis
Solari, A., Uitdehaag, B. M. J., Giuliani, G., Pucci, E., & Taus, C. (2002) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001330 · Cited by 78
Searching Twitter: Separating the Tweet from the Chaff
Hurlock, J., & Wilson, M. L. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14117 · Cited by 79
“See: http://www.bbc.co.uk/proms/2010/ the three coders achieved a Fleiss’s kappa (Fleiss, 1971) of 0.73 for the not-useful tweets.”
BackgroundMethodInfluentialVariance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of Raters
Gwet, K. L. (2008) · Psychometrika · doi:10.1007/s11336-007-9054-8 · Cited by 91
“Fleiss (1971) proposed one of the most popular multiple-rater agreement coefficients in use today.”
BackgroundMethodInfluentialAutomatic term and relation extraction for medical question answering system
Fahmi, I. (2009) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 91
Bewersdorff, A., Seßler, K., Baur, A., Kasneci, E., & Nerdel, C. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100177 · Cited by 76
“The ratings are compared between human raters and the AI-generated analyses by metrics common in the field of AI (Accuracy) and in the field of social sciences (Cohens Kappa, Cohen, 1960; Fleiss Kappa, Fleiss, 1971 and Gwet’s AC1, Gwet, 2014).”
MethodInfluentialGage, E. A. (2012) · Social Science & Medicine · doi:10.1016/j.socscimed.2012.09.001 · Cited by 94
“Inter-coder reliability was extremely high (Fleiss’ kappa for specific items ranged from .91 to 1), and any discrepancies were discussed and resolved at coding meetings (Fleiss, 1971).”
BackgroundUser Variability and IR System Evaluation
Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767728 · Cited by 73
“664, measured by Fleiss’ κ [12], a statistic that measures agreement across multiple raters and corrects for agreement by chance.”
MethodAffirmative Cue Words in Task-Oriented Dialogue
Gravano, A., Hirschberg, J., & Beňuš, Š. (2012) · Computational Linguistics · doi:10.1162/coli_a_00083 · Cited by 77
“Inter-labeler reliability was measured by Fleiss’s κ (Fleiss 1971) as Substantial at 0.”
InfluentialPrediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning
Tian, F., Liu, D., Wei, N., Fu, Q., Sun, L., Liu, W., Sui, X., Tian, K., Nemeth, G., Feng, J., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02915-w · Cited by 83
“We used Fleiss’ kappa ( κ ) 58,59 to measure inter-rater reliability when including multiple raters and more than two categories, which was calculated according to where p o is the observed agreement rate and p e the expected agreement rate.”
MethodDefining and conceptualizing equity and justice in climate adaptation
Walker, S. E., Smith, E. A., Bennett, N., Bannister, E., Narayana, A., Nuckols, T., Velez, K. P., Wrigley, J., & Bailey, K. M. (2024) · Global Environmental Change · doi:10.1016/j.gloenvcha.2024.102885 · Cited by 122
Imaginary companions in adolescence: sign of a deficient or positive development?
Seiffge‐Krenke, I. (1997) · Journal of Adolescence · doi:10.1006/jado.1996.0072 · Cited by 78
Ferrara, G., Argenziano, G., Soyer, H. P., Corona, R., Sera, F., Brunetti, B., Cerroni, L., Chimenti, S., Shabrawi‐Caelen, L. E., Ferrari, A., et al. (2002) · Cancer · doi:10.1002/cncr.10768 · Cited by 81
“In the current study, in which all eight observers judged the fixed set of 107 lesions, the formula proposed by Schouten(25) gives the same kappa values as the procedure proposed by Fleiss.(26) The standard error of kappa was calculated by the standard jack knife method.”
MethodInfluentialPragmatic aspects of temporal accommodation in turn-taking
Beňuš, Š., Gravano, A., & Hirschberg, J. (2011) · Journal of Pragmatics · doi:10.1016/j.pragma.2011.05.011 · Cited by 77
“Inter-labeler reliability was measured by Fleiss' kappa (Fleiss, 1971) at 0.69.”
MethodInfluential
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.