Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 601–700 of 1,118
Response Generation by Context-Aware Prototype Editing
Wu, Y., Wei, F., Huang, S., Wang, Y., Li, Z., & Zhou, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017281 · Cited by 110
Statistics in Corpus Linguistics: A Practical Guide
Březina, V. (2018) · Cited by 110
Policy Entrepreneurs, Legislators, and Agenda Setting: Information and Influence
Anderson, S. E., DeLeo, R. A., & Taylor, K. N. (2019) · Policy Studies Journal · doi:10.1111/psj.12331 · Cited by 118
The Art of Feature Engineering: Essentials for Machine Learning
Duboue, P. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108671682 · Cited by 171
The General Health Questionnaire: Reliability and validity for Australian youth
Winefield, H. R., Goldney, R. D., Winefield, A. H., & Tiggemann, M. (1989) · Australian & New Zealand Journal of Psychiatry · doi:10.3109/00048678909062592 · Cited by 110
AUSUM: approach for unsupervised bug report summarization
Mani, S., Catherine, R., Sinha, V. S., & Dubey, A. (2012) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2393596.2393607 · Cited by 135
“We also performed the Kappa test [14] to measure the level of agreement between the two members, as typically each individual can summarize the bug report in their own ways.”
MethodChng, S. M., Petersen, E. T., Zimine, I., Sitoh, Y., Lim, C. C. T., & Golay, X. (2008) · Stroke · doi:10.1161/strokeaha.108.520593 · Cited by 116
Internal Carotid Arterial Stenosis: CT Angiography with Volume Rendering
Leclerc, X., Godefroy, O., Lucas, C., Benhaim, J.-F., Michel, T. S., Leys, D., & Pruvo, J. (1999) · Radiology · doi:10.1148/radiology.210.3.r99fe46673 · Cited by 110
Lampasona, V., Schlosser, M., Mueller, P. W., Williams, A. J., Wenzlau, J. M., Hutton, J. C., Achenbach, P., & Laboratories, P. (2011) · Clinical Chemistry · doi:10.1373/clinchem.2011.170662 · Cited by 109
Resch, B., Summa, A., Zeile, P., & Strube, M. (2016) · Urban Planning · doi:10.17645/up.v1i2.617 · Cited by 124
“Concretely, we use the Fleiss Kappa index (Fleiss, 1971), which generalises the original Cohen’s Kappa (Cohen, 1960) to more than two annotators and classes.”
MethodUser Intent, Behaviour, and Perceived Satisfaction in Product Search
Su, N., He, J., Liu, Y., Zhang, M., & Ma, S. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159714 · Cited by 108
“The Fleiss’ κ [8] is 0.756 among three annotators.”
BackgroundInfluentialTechnologically scaffolded atypical cognition: the case of YouTube’s recommender system
Alfano, M., Fard, A. E., Carter, J. A., Clutton, P., & Klein, C. (2021) · Synthese · doi:10.1007/s11229-020-02724-x · Cited by 107
Nantee, N., & Sureeyatanapas, P. (2021) · Benchmarking An International Journal · doi:10.1108/bij-11-2020-0583 · Cited by 132
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Cited by 109
Huber, O. (1980) · Acta Psychologica · doi:10.1016/0001-6918(80)90031-1 · Cited by 109
QUESTIONNAIRE ASSESSMENT OF LIFETIME AND RECENT EXPOSURE TO ENVIRONMENTAL TOBACCO SMOKE
Coultas, D., Peake, G. T., & SAMETI, J. M. (1989) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a115340 · Cited by 109
A Comparison of Two Structured Diagnostic Interviews: CIDI and SCAN
Andrews, G., Peters, L., Guzman, A.-M., & Bird, K. (1995) · Australian & New Zealand Journal of Psychiatry · doi:10.3109/00048679509075901 · Cited by 108
The Music Information Retrieval Evaluation eXchange: Some Observations and Insights
Downie, J. S., Ehmann, A. F., Bay, M., & Jones, M. C. (2010) · Studies in computational intelligence · doi:10.1007/978-3-642-11674-2_5 · Cited by 108
“It is a measure of inter-grader reliability for nominal data and is based upon Cohen’s two-grader reliability Kappa but is intended for use with an arbitrary number of graders [8].”
MethodConrad, C. R., Haglund, J., & Moore, W. H. (2013) · International Studies Perspectives · doi:10.1111/j.1528-3585.2012.00471.x · Cited by 108
Affect Analysis Model: novel rule-based approach to affect sensing from text
Neviarouskaya, A., Prendinger, H., & Ishizuka, M. (2010) · Natural Language Engineering · doi:10.1017/s1351324910000239 · Cited by 138
“We faced the difficulty of employing Fleiss’ Kappa coefficient (Fleiss 1971) to measure inter-rater agreement here, as the important requirement of using it is that each entry needs to be assigned to only one of possible categories.”
MethodInfluentialLearning from the past: answering new questions with past answers
Shtok, A., Dror, G., Maarek, Y., & Szpektor, I. (2012) · International World Wide Web Conference (WWW) · doi:10.1145/2187836.2187939 · Cited by 126
“To assess the quality of the workers, we calculated the inter-worker agreement in each category using Fleiss’ kappa [13, 26].”
MethodAn Ensemble of Retrieval-Based and Generation-Based Human-Computer Conversation Systems
Song, Y., Li, C.-T., Nie, J.-Y., Zhang, M., Zhao, D., & Yan, R. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/609 · Cited by 108
Kappa coefficients in epidemiology: An appraisal of a reappraisal
Kraemer, H. C., & Bloch, D. A. (1988) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(88)90032-7 · Cited by 108
Swiontkowski, M. F., Sands, A. K., Agel, J. A., Diab, M. G., Schwappach, J. R., & Kreder, H. J. (1997) · Journal of Orthopaedic Trauma · doi:10.1097/00005131-199710000-00002 · Cited by 108
A comparative case study of sustaining quality as a competitive advantage
Su, H., Linderman, K., Schroeder, R. G., & Van de Ven, A. H. (2014) · Journal of Operations Management · doi:10.1016/j.jom.2014.09.003 · Cited by 107
Health Outcomes for Living Kidney Donors with Isolated Medical Abnormalities: A Systematic Review
Young, A., Storsley, L. J., Garg, A. X., Treleaven, D. J., Nguan, C. Y. C., Cuerden, M. S., & Karpinski, M. (2008) · American Journal of Transplantation · doi:10.1111/j.1600-6143.2008.02339.x · Cited by 107
“Statistical analysis Interrater reliability was assessed using Fleiss’ generalized j, which describes the chance-corrected agreement among more than two raters (16,17).”
BackgroundMethodVerbal Interaction Sequences and Group Mood: Exploring the Role of Team Planning Communication
Lehmann‐Willenbrock, N., Meyers, R. A., Kauffeld, S., Neininger, A., & Henschel, A. (2011) · Small Group Research · doi:10.1177/1046496411398397 · Cited by 124
Model transformation intents and their properties
Lúcio, L., Amrani, M., Juergen Dingel, A., Lambers, L., Salay, R., Selim, G. M. K., Syriani, E., & Wimmer, M. (2014) · Software & Systems Modeling · doi:10.1007/s10270-014-0429-x · Cited by 107
“For Q2, we used the statistical measure, called Fleiss’ kappa [39], to assess the inter-rater reliability of agreement.”
MethodIncorporating User Expectations and Behavior into the Measurement of Search Effectiveness
Moffat, A., Bailey, P., Scholer, F., & Thomas, P. (2017) · ACM Transactions on Information Systems · doi:10.1145/3052768 · Cited by 107
Click communication in wild harbour porpoises (Phocoena phocoena)
Sørensen, P. M., Wiśniewska, D. M., Jensen, F. H., Johnson, M. P., Teilmann, J., & Madsen, P. T. (2018) · Scientific Reports · doi:10.1038/s41598-018-28022-8 · Cited by 107
Human movement and ergonomics: An industry-oriented dataset for collaborative robotics
Maurice, P., Malaisé, A., Amiot, C., Paris, N., Richard, G.-J., Rochel, O., & Ivaldi, S. (2019) · The International Journal of Robotics Research · doi:10.1177/0278364919882089 · Cited by 128
“We computed Fleiss’ κ to assess interrater reliability (Fleiss 1971).”
García, I., Pacheco, C., Méndez, F., & Calvo-Manzano, J. A. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22304 · Cited by 124
“The inter ‐ rater agreement depicts the consistency of the data extraction among the analyzed studies when two or more researchers evaluate each study [44].”
Background`I make up a silly name': Understanding Children's Perception of Privacy Risks Online
Zhao, J., Wang, G., Dally, C., Slovak, P., Edbrooke-Childs, J., Van Kleek, M., & Shadbolt, N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300336 · Cited by 105
“These codes were then applied to the second (yet unseen) set of transcriptions by the same set of researchers, and Fleiss’ kappa [28] (0.83) was computed to assess inter-coder reliability.”
MethodInfluentialBiopsychobehavioral correlates of insomnia. IV: Diagnosis based on DSM- III
(1984) · American Journal of Psychiatry · doi:10.1176/ajp.141.3.357 · Cited by 105
A Theory of Vagueness and Privacy Risk Perception
Bhatia, J., Breaux, T. D., Reidenberg, J. R., & Norton, T. B. (2016) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2016.20 · Cited by 110
“We applied Fleiss’ Kappa, an inter-rater agreement statistic [23], to the annotations-vagueness category mappings.”
MethodBry, L. J., Chou, T., Miguel, E., & Comer, J. S. (2018) · Behavior Therapy · doi:10.1016/j.beth.2017.07.008 · Cited by 106
“Overall Kappa reliability coefficients ranged from k=0.71–1.00 on individual codes and an overall reliability of k=0.94 indicating strong reliability (Fleiss, 1971).”
BackgroundInfluentialOverview of the 8th Workshop on Asian Translation
Nakazawa, T., Nakayama, H., Ding, C., Dabre, R., Higashiyama, S., Mino, H., Goto, I., Pa, W. P., Kunchukuttan, A., Parida, S., et al. (2021) · Workshop on Asian Translation (WAT) · doi:10.18653/v1/2021.wat-1.1 · Cited by 111
Al-Awidi, H. M., & Alghazo, I. (2012) · Educational Technology Research and Development · doi:10.1007/s11423-012-9239-4 · Cited by 144
“Fleiss’s kappa statistic (1971), was used to measure agreement among raters.”
MethodNeurofilament ELISA validation
Petzold, A., Altintas, A., Andreoni, L., Bartos, A., Berthele, A., Blankenstein, M. A., Buee, L., Castellazzi, M., Cepok, S., Comabella, M., et al. (2010) · Journal of Immunological Methods · doi:10.1016/j.jim.2009.09.014 · Cited by 105
“The inter–laboratory agreement was assessed using kappa statistics for multiple raters and multiple samples [15] and rated as described [16].”
MethodArtificial intelligence tools expand scientists' impact but contract science's focus
Hao, Q., Xu, F., Li, Y., & Evans, J. (2026) · Nature · doi:10.1038/s41586-025-09922-y · Cited by 117
“37,38).”
A Longitudinal Study of Superior-Subordinate Communication During Job Transfers
KRAMER, M. W. (1995) · Human Communication Research · doi:10.1111/j.1468-2958.1995.tb00361.x · Cited by 105
Tsuda, H., Akiyama, F., Kurosumi, M., Sakamoto, G., Watanabe, T., & Section, J. N. S. A. S. O. B. C. (.-B. P. (1998) · Japanese Journal of Clinical Oncology · doi:10.1093/jjco/28.8.486 · Cited by 105
Ross, K. (1999) · Aphasiology · doi:10.1080/026870399402235 · Cited by 105
The Prosody of Backchannels in American English
Stefan Benus, A., Gravano, A., & Hirschberg, J. (2007) · doi:10.7916/d8ww7s1p · Cited by 82
“Interlabeler reliability was measured by Fleiss' κ [3] at 0.74 for the CARDS, 0.63 for the OBJECTS, and 0.69 for the whole corpus, where values between 0.6 and 0.8 correspond to substantial agreement.”
MethodInfluentialAriji, Y., Sakuma, S., Izumi, M., Sasaki, J., Kurita, K., Ogi, N., Nojiri, M., Nakagawa, M., Takenaka, M., 勝瀬, 誠., et al. (2004) · Oral Surgery Oral Medicine Oral Pathology Oral Radiology and Endodontology · doi:10.1016/j.tripleo.2004.06.068 · Cited by 104
Assessment of the Predictive Validity of Etiologic Stroke Classification
Arsava, E. M., Helenius, J., Avery, R., Sorgun, M. H., Kim, G., Pontes‐Neto, O. M., Park, K., Rosand, J. M., Vangel, M. G., & Ay, H. (2017) · JAMA Neurology · doi:10.1001/jamaneurol.2016.5815 · Cited by 105
Bug Characteristics in Blockchain Systems: A Large-Scale Empirical Study
Wan, Z., Lo, D., Xia, X., & Cai, L. (2017) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.59 · Cited by 108
“We use Fleiss Kappa [20] to measure the agreement between the two labelers.”
KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation
Zhou, H., Zheng, C., Huang, K., Huang, M., & Zhu, X. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.635 · Cited by 104
Wagner, M., Müller-Stich, B. P., Kisilenko, A., Tran, D., Heger, P., Mündermann, L., Lubotsky, D. M., Müller, B., Davitashvili, T., Capek, M., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102770 · Cited by 171
Kaasen, A., Helbig, A., Malt, U. F., Næs, T., Skari, H., & Haugen, G. N. (2010) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2010.02622.x · Cited by 121
Zarse, C. A., Hameed, T., Jackson, M. E., Pishchalnikov, Y. A., Lingeman, J. E., McAteer, J. A., & Williams, J. C. (2007) · Urological Research · doi:10.1007/s00240-007-0104-6 · Cited by 104
“1 (SAS, Cary, NC, USA), or as Fleiss’ for more than two observers [22].”
BackgroundCarlson, G. W., Page, A. L., Peters, K. K., Ashinoff, R. L., Schaefer, T. G., & D, A. L. M. (2008) · Annals of Plastic Surgery · doi:10.1097/sap.0b013e31815b6ced · Cited by 104
“Inter-rater reliability with more than 2 raters was calculated for each of the subscales, the total sum of the subscales, and the global cosmetic score using the Fleiss’ statistic.(6) The strength of agreement was classified based on the score, 0.”
MethodFox, E. (1993) · JAMA · doi:10.1001/jama.1993.03510210064029 · Cited by 104
Lampasona, V., Pittman, D. L., Williams, A. J., Achenbach, P., Schlosser, M., Akolkar, B., Winter, W. E., Laboratories, P., Watson, K., Weets, I., et al. (2019) · Clinical Chemistry · doi:10.1373/clinchem.2019.304196 · Cited by 103
“interlaboratory comparison of glutamic acid decarboxylase autoantibody 2 assay performance 3 Lampasona Va*^, Pittman DLb^, Williams AJc, Achenbach Pd, Schlosser Mef, Akolkar Bg, 4 Winter WEb, and participating laboratories 5 aSan Raffaele Diabetes Research Institute, San Raffaele Scientific Institute, Milan, Italy; 6…”
BackgroundMethodInfluentialOffensive, aggressive, and hate speech analysis: From data-centric to human-centered approach
Kocoń, J., Figas, A., Gruza, M., Puchalska, D., Kajdanowicz, T., & Kazienko, P. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102643 · Cited by 103
Quantifying Political Leaning from Tweets, Retweets, and Retweeters
Wong, F. M. F., Tan, C. W., Sen, S., & Chiang, M. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2553667 · Cited by 136
“430 [14], a moderate level of agreement [23].”
BackgroundLukito, J., Suk, J., Zhang, Y., Doroshenko, L., Kim, S. J., Su, M.-H., Xia, Y., Freelon, D., & Wells, C. (2020) · The International Journal of Press/Politics · doi:10.1177/1940161219895215 · Cited by 103
“After three rounds of training, intercoder reliability was established using a 15 percent (n = 35) random sample; Fleiss’ kappa (FK; Fleiss 1971) for all variables was above 0.7 (see Supplemental Appendix C and D for codebook and example tweets).”
MethodInfluentialA Large-Scale Chinese Short-Text Conversation Dataset
Wang, Y., Ke, P., Zheng, Y., Huang, K., Jiang, Y., Zhu, X., & Huang, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60450-9_8 · Cited by 167
COLD: A Benchmark for Chinese Offensive Language Detection
Deng, J., Zhou, J., Sun, H., Zheng, C., Mi, F., Meng, H. M. L., & Huang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.796 · Cited by 226
Phonological Words and Stuttering on Function Words
Au‐Yeung, J., Howell, P., & Pilgrim, L. (1998) · Journal of Speech Language and Hearing Research · doi:10.1044/jslhr.4105.1019 · Cited by 107
“The agreement between the two judges on content/function word was 98% giving a kappa coefficient of .96, which is much higher than chance (Fleiss, 1971).”
BackgroundCan Spanish Be Simpler? LexSiS: Lexical Simplification for Spanish
Bott, S., Rello, L., Drndarević, B., & Saggion, H. (2012) · International Conference on Computational Linguistics · Cited by 105
Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis.
Orwin, R. G., & Cordray, D. S. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.97.1.134 · Cited by 102
Configurations of human resource management practices: a model and test of internal fit
Verburg, R. M., Den Hartog, D. N., & Koopman, P. L. (2007) · The International Journal of Human Resource Management · doi:10.1080/09585190601102349 · Cited by 101
“In addition to Cohen's kappa the ‘multi-rater kappa’ (κ v ) as developed by Fleiss ( 1971 ) was computed (see Siegel and Castellan, 1988 ).”
BackgroundMethodInfluentialFatty infiltration of the supraspinatus: A reliability study
Williams, M. D., Lädermann, A., Mélis, B., Barthélémy, R., & Walch, G. (2009) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2008.12.014 · Cited by 102
“The intraobserver and interobserver reliability of the Goutallier scores were determined for each set of images by calculating multijudge k coefficient of agreement.(2,3) A second analysis was then made to assess reviewer agreement using a modified 3-grade scale of the Goutallier classification as described by Fuchs e…”
MethodIssues in the Use of Kappa to Estimate Reliability
Soeken, K. L., & Prescott, P. A. (1986) · Medical Care · doi:10.1097/00005650-198608000-00008 · Cited by 102
Predicting political preference of Twitter users
Makazhanov, A., Rafiei, D., & Waqar, M. (2014) · Social Network Analysis and Mining · doi:10.1007/s13278-014-0193-5 · Cited by 110
“The rate of inter-annotator agreement calculated as Fleiss’ kappa [5] was 0.”
MethodYom‐Tov, E., & Gabrilovich, E. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2614 · Cited by 109
“The interannotator agreement estimated using the Fleiss’ kappa statistic [25] was 0.”
MethodGood News? A Longitudinal Analysis of Newspaper Portrayals of Mental Illness in Canada 2005 to 2015
Whitley, R., & Wang, J. (2017) · The Canadian Journal of Psychiatry · doi:10.1177/0706743716675856 · Cited by 101
The Acoustic Breathiness Index (ABI): A Multivariate Acoustic Model for Breathiness
Latoszek, B. B. V., Maryn, Y., Gerrits, E., & Bodt, M. S. D. (2017) · Journal of Voice · doi:10.1016/j.jvoice.2016.11.017 · Cited by 127
Oberländer, A. M., Röglinger, M., Rosemann, M., & Kees, A. (2018) · European Journal of Information Systems · doi:10.1080/0960085x.2017.1387714 · Cited by 101
“Cohen’s Kappa coefficient (Cohen, 1960) Fleiss’ Kappa coefficient (Fleiss, 1971)”
BackgroundAL: An Adaptive Learning Support System for Argumentation Skills
Wambsganss, T., Niklaus, C., Cetto, M., Söllner, M., Handschuh, S., & Leimeister, J. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376732 · Cited by 101
“Moreover, we calculated percentage agreement and two chance-corrected measures, multi-π [20] and Krippendorff’s α [32], to evaluate the annotators’ agreement in terms of the selected category of an argument component for a given sentence.”
MethodPolhemus, A., Delgado-Ortiz, L., Brittain, G., Chynkiamis, N., Salis, F., Gaßner, H., Gross, M., Kirk, C., Rossanigo, R., Taraldsen, K., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00513-5 · Cited by 101
Jensen, M. D., Ormstrup, T. E., Vagn-Hansen, C., Ostergaard, L. L., & Rafaelsen, S. R. (2010) · Inflammatory Bowel Diseases · doi:10.1002/ibd.21534 · Cited by 101
Selecting single items to measure doubly concrete constructs: A cautionary tale
Sarstedt, M., Diamantopoulos, A., Salzberger, T., & Baumgartner, P. (2016) · Journal of Business Research · doi:10.1016/j.jbusres.2015.12.004 · Cited by 100
Henning, G., Stenling, A., Bielak, A. A. M., Bjälkebring, P., Gow, A. J., Kivi, M., Terrera, G. M., Johansson, B. E. A., & Lindwall, M. (2020) · Aging & Mental Health · doi:10.1080/13607863.2019.1709156 · Cited by 127
“The inter-rater reliability (Fleiss ’ Kappa, Fleiss, 1971) was high and significant ( j ¼ .71, p .001) reflecting substantial agreement among the raters (see Appendix A).”
BackgroundLanguage Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
Ji, H., Ke, P., Huang, S., Wei, F., Zhu, X., & Huang, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.54 · Cited by 95
Watts, F. M., & Finkenstaedt-Quinn, S. A. (2021) · Chemistry Education Research and Practice · doi:10.1039/d1rp00007a · Cited by 120
“…limitations: weighted kappa allows for ratio or interval coding, Fleiss’ kappa and Light’s kappa each allow for more than two researchers, and fuzzy kappa allows for multiple codes to be applied to each unit of analysis (Cohen, 1968; Fleiss, 1971; Light, 1971; Kirilenko and Stepchenkova, 2016).”
A query expansion method for retrieving online BIM resources based on Industry Foundation Classes
Gao, G., Liu, Y.-S., Wang, M., Gu, M., & Yong, J. (2015) · Automation in Construction · doi:10.1016/j.autcon.2015.04.006 · Cited by 103
“In particular, the inter-rater agreement (or inter-rater reliability) of document ranking can be assessed by some statistical measures such as Fleiss' kappa [54].”
MethodSantos, C., Bernardes, J., Ayres‐de‐Campos, D., Costa, A., & Amorim‐Costa, C. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.010 · Cited by 99
“The proportions of overall agreement were obtained with generalized formulas for any number of raters with potentially different numbers of raters or different raters for each case [12], and the 95% CIs were obtained using nonparametric bootstrap [13].”
MethodPhillips, J. C., Ingram, K. M., Smith, N. G., & Mindes, E. J. (2003) · The Counseling Psychologist · doi:10.1177/0011000002239398 · Cited by 119
“To assess interrater reliability, we randomly selected 20% of the articles (n = 24) and calculated unweighted, multirater kappa coefficients (Fleiss, 1971) for items that required some degree of judgment based on data from the three raters.”
MethodQuestions are content: A taxonomy of questions in a microblogging environment
Efron, M., & Winget, M. (2010) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504701208 · Cited by 112
Volunteer decisions (not) to leave: Reasons to quit versus functional motives to stay
Willems, J., Huybrechts, G., Jegers, M., Vantilborgh, T., Bidee, J., & Pepermans, R. (2012) · Human Relations · doi:10.1177/0018726712442554 · Cited by 123
“We use Fleiss’ kappa to report on the expert ratings as it enables us to assess agreement (or accordance) of two or more different categorical classifications, with the same number of categories per classification (Fleiss, 1971; Gamer et al., 2010).”
MethodTaxonomy induction based on a collaboratively built knowledge repository
Ponzetto, S. P., & Strube, M. (2011) · Artificial Intelligence · doi:10.1016/j.artint.2011.01.003 · Cited by 120
“We use one variant, Fleiss’ kappa [30], which is a generalization of Scott’s π for more than two annotators.”
MethodA Population-Based Study of Inferior Vena Cava Filters in Patients With Acute Venous Thromboembolism
Spencer, F. A., Bates, S. M., Goldberg, R. J., Lessard, D., Emery, C., Glushchenko, A., Gore, J. M., & White, R. H. R. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.272 · Cited by 105
“The results were analyzed using the Fleiss nominal scale agreement among many raters.(7)”
Gold-standard for computer-assisted morphological sperm analysis
Chang, V., García, A., Hitschfeld, N., & Härtel, S. (2017) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2017.03.004 · Cited by 116
Factors Affecting Fertility Decision-Making Among Transgender Adolescents and Young Adults
Chen, D., Kyweluk, M. A., Sajwani, A., Gordon, E. J., Johnson, E. K., Finlayson, C. A., & Woodruff, T. K. (2019) · LGBT Health · doi:10.1089/lgbt.2018.0250 · Cited by 108
“91 (13 excerpts), indicating excellent interrater reliability.(24) Cross-case analyses were conducted using Fisher’s exact tests to examine whether there were significant patterns in identified themes based on participants’ birth-assigned sex and FP consult and completion status.”
MethodInterviewer Questions and Content Analysis of Children's Statements of Sexual Abuse
Craig, R. A., Scheibe, R., Raskin, D. C., Kircher, J. C., & Dodd, D. H. (1999) · Applied Developmental Science · doi:10.1207/s1532480xads0302_2 · Cited by 99
Learning Sentiment from Students’ Feedback for Real-Time Interventions in Classrooms
Altrabsheh, N., Cocea, M., & Fallahkhair, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11298-5_5 · Cited by 101
“6%, the Fleiss kappa [7] was 0.”
Analyzing Argumentative Discourse Units in Online Interactions
Ghosh, D., Muresan, S., Wacholder, N., Aakhus, M. A., & Mitsui, M. R. (2014) · Workshop on Argument Mining (ArgMining) · doi:10.3115/v1/w14-2106 · Cited by 117
Hirasawa, A., Wakao, N., Kamiya, M., Takeuchi, M., Kawanami, K., Murotani, K., Matsuo, T., & Deie, M. (2016) · Journal of Orthopaedic Science · doi:10.1016/j.jos.2016.02.001 · Cited by 106
“Chi-square test, Cohen kappa [10] for intra-observer error and Fleiss kappa [11] for inter-observer error were used for the comparison of subject data and results.”
Post-mortem histopathology underlying β-amyloid PET imaging following flutemetamol F 18 injection
Ikonomovic, M. D., Buckley, C. J., Heurling, K., Sherwin, P., Jones, P. A., Zanette, M., Mathis, C. A., Klunk, W. E., Chakrabarty, A., Ironside, J., et al. (2016) · Acta Neuropathologica Communications · doi:10.1186/s40478-016-0399-z · Cited by 98
“The Fleiss coefficient Pi was used to measure inter-reader agreement [Pi = 1/n(n-1)*(A 2 +N(2)-n), where n is the number of readers, A is the number calling the case abnormal, and N is the number of readers calling the case negative] [18].”
MethodReproducibility of a four-point clinical severity score for glabellar frown lines
Honeck, P. R., Weiss, C., Sterry, W., Rzany, B., & for the Gladys study group (2003) · British Journal of Dermatology · doi:10.1046/j.1365-2133.2003.05436.x · Cited by 100
Predictors of Outcome in an Open Label, Therapeutic Trial of High-Dose Omeprazole in Laryngitis
Williams, R. B. H., Szczesniak, M. M., Maclean, J., Brake, H., Cole, I. E., & Cook, I. J. (2004) · The American Journal of Gastroenterology · doi:10.1111/j.1572-0241.2004.04151.x · Cited by 106
“To obtain a more precise index of concordance among observers for each evaluation method, a κ-statistic was derived (27, 28).”
MethodThe reliability of psychiatric and psychological diagnosis
Matarazzo, J. D. (1983) · Clinical Psychology Review · doi:10.1016/0272-7358(83)90008-9 · Cited by 98
Examiner alignment and assessment in clinical periodontal research
Hefti, A. F., & Preshaw, P. M. (2012) · Periodontology 2000 · doi:10.1111/j.1600-0757.2011.00436.x · Cited by 97
Fielding, S., Porteous, T., Ferguson, J., Maskrey, V., Blyth, A., Paudyal, V., Barton, G., Holland, R., Bond, C. M., & Watson, M. C. (2015) · Family Practice · doi:10.1093/fampra/cmv003 · Cited by 100
“Inter-rater agreement was calculated using pairwise Cohen’s kappa (κ) for each panel member with each other panel member and across the whole panel using multiple kappa (19,20).”
MethodKhurum, M., Gorschek, T., & Wilson, M. (2012) · Journal of Software Evolution and Process · doi:10.1002/smr.1560 · Cited by 126
“The Fleiss’ Kappa [41] value showed a reasonable agreement (0.”
BackgroundFisher, C. G., Versteeg, A. L., Schouten, R., Boriani, S., Varga, P. P., Rhines, L. D., Heran, M. K. S., Kawahara, N., Fourney, D. R., Reynolds, J., et al. (2014) · American Journal of Roentgenology · doi:10.2214/ajr.13.12269 · Cited by 100
“Statistical Analysis Interobserver agreement (Fleiss’ kappa) for the radiologists and surgeons and intraobserver reproducibility (Cohen’s kappa) among the radiologists were calculated for the total SINS score, the three clinical categories, and the binary scale [11, 12].”
MethodLudoliteracy: Defining, Understanding, and Supporting Games Education
Zagal, J. P. (2024) · KiltHub Repository · doi:10.1184/r1/6686828 · Cited by 98
Assessment of the trabecular pattern before endosseous implant treatment
Lindh, C., Petersson, A. R., & Rohlin, M. (1996) · Oral Surgery Oral Medicine Oral Pathology Oral Radiology and Endodontology · doi:10.1016/s1079-2104(96)80363-5 · Cited by 117
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.