Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 1,001–1,100 of 1,118
Vavricka, S. R., Storck, C. A., Wildi, S. M., Tutuian, R. I., Wiegand, N., Rousson, V., Fruehauf, H., Müllhaupt, B., & Fried, M. (2007) · The American Journal of Gastroenterology · doi:10.1111/j.1572-0241.2007.01145.x · Cited by 73
Unpacking Customer Rage Elicitation: A Dynamic Model
Surachartkumtonkun, J., McColl‐Kennedy, J. R., & Patterson, P. G. (2014) · Journal of Service Research · doi:10.1177/1094670514556275 · Cited by 86
The Unified and Holistic Method Gamma (γ) for Inter-Annotator Agreement Measure and Alignment
Mathet, Y., Widlöcher, A., & Métivier, J.-P. (2015) · Computational Linguistics · doi:10.1162/coli_a_00227 · Cited by 151
Mixed methods student evaluation of an online systemic human anatomy course with laboratory
Attardi, S. M., Choi, S., Barnett, J., & Rogers, K. A. (2015) · Anatomical Sciences Education · doi:10.1002/ase.1584 · Cited by 75
Answering real-world clinical questions using large language model based systems
Low, Y. S., Jackson, M. L., Hyde, R. J., Brown, R. E., Sanghavi, N. M., Baldwin, J. D., Pike, C. W., Muralidharan, J., Hui, G., Alexander, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.00541 · Cited by 68
“We calculated the inter-rater agreement on response category using Krippendorff's Alpha 27 and on the best and actionable metrics using Fleiss’ Kappa 28 .”
MethodInfluentialGarali, I., Adanyeguh, I. M., Ichou, F., Perlbarg, V., Seyer, A., Colsch, B., Moszer, I., Guillemot, V., Durr, A., Mochel, F., et al. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbx060 · Cited by 68
The Hypnotic Induction Profile: Normative Observations, Reliability and Validity
Stern, D. B., Spiegel, H., & Nee, J. C. M. (1978) · American Journal of Clinical Hypnosis · doi:10.1080/00029157.1978.10403967 · Cited by 67
A Standardized Lingual Tonsil Grading System: Interexaminer Agreement
Friedman, M., Yalamanchali, S., Gorelick, G., Joseph, N. J., & Hwang, M. S. (2015) · Otolaryngology–Head and Neck Surgery · doi:10.1177/0194599815568970 · Cited by 65
“The Fleiss kappa (k) statistic was used to evaluate interobserver agreement.(9) The k value was interpreted according to guidelines by Landis and Koch.”
BackgroundMethodMacNeil, M., Koch, M., Kuspinar, A., Juzwishin, D., Lehoux, P., & Stolee, P. (2019) · Health Policy · doi:10.1016/j.healthpol.2018.09.018 · Cited by 65
Antoine, J.-Y., Villaneau, J., & Lefeuvre, A. (2014) · Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1058 · Cited by 87
Rameckers, S. A., Verhoef, R. E. J., Grasman, R. P. P. P., Cox, W. R., van Emmerik, A. A. P., Engelmoer, I. M., & Arntz, A. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10235622 · Cited by 63
“The interrater agreement, Fleiss’ kappa [34],”
Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models
Than, N., Fan, L., Law, T., Nelson, L. K., & McCall, L. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339188 · Cited by 63
“To do so, we used standard “ interrater reliability ” measures to calculate two forms of prompt and model agreement from our output: inter-prompt agreement , de fi ned as the degree of agreement within one LLM across prompts, and intermodel agreement , de fi ned as the degree of agreement within one prompt across differ…”
InfluentialDickinson, D., Pratt, D. N., Giangrande, E. J., Grunnagle, M., Orel, J., Weinberger, D. R., Callicott, J. H., & Berman, K. F. (2018) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx039 · Cited by 62
The effects of information and social conformity on opinion change
Mallinson, D. J., & Hatemi, P. K. (2018) · PLOS ONE · doi:10.1371/journal.pone.0196600 · Cited by 62
Acoustic and Lexical Sentiment Analysis for Customer Service Calls
Li, B., Dimitriadis, D., & Stolcke, A. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683679 · Cited by 62
“The metric used is Fleiss’ kappa [11], a generalization of Cohen’s kappa for multiple labelers in which each pair of labels for an item is treated as a potential agreement or disagreement.”
MethodPersonality disorders in veterans with posttraumatic stress disorder and depression
Dunn, N. J., Yanasak, E., Schillaci, J., Simotas, S., Rehm, L. P., Souchek, J., Menke, T., Ashton, C., & Hamilton, J. D. (2004) · Journal of Traumatic Stress · doi:10.1023/b:jots.0000014680.54051.50 · Cited by 61
“(The PTSD reliability was based on 20 interviews rather than 23, because of technical difficulties.) For interrater reliability of each diagnosis, we calculated both the percent agreement and Cohen’s kappa, using the Fleiss (1971) algorithm for different pairs of raters evaluating each participant.”
MethodMasakhaNER: Named entity recognition for African languages
(2021) · HAL · doi:10.1162/tacl · Cited by 61
ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases
Mehnen, L., Gruarin, S., Vasileva, M., & Knapp, B. (2023) · doi:10.1101/2023.04.20.23288859 · Cited by 60
“For the diagnoses agreement hypothesis-test between 3.5, 4 and the correct diagnosis, we used Fleiss' kappa [10], which is an extension of Scott's pi statistic [11].”
MethodInfluentialExternal validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis
Rockenschaub, P., Akay, E. M., Carlisle, B. G., Hilbert, A., Wendland, J., Meyer-Eschenbach, F., Näher, A.-F., Frey, D., & Madai, V. I. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02830-7 · Cited by 58
“Agreement between authors was assessed using Fleiss’ Kappa [22].”
Halliday, G., Ng, T., Rodriguez, M., Harding, A., Blumbergs, P., Evans, W., Fabian, V., Fryer, J., Gonzales, M., Harper, C., et al. (2002) · Acta Neuropathologica · doi:10.1007/s00401-002-0529-5 · Cited by 57
“Percentage agreement (sensitivity, Table 2) and positive predictive values (PPV, Table 2), in addition to the extension of the kappa index for multiple raters [ 13 ], were used to determine the level of diagnostic agreement.”
MethodToward a semantic granularity model for domain-specific information retrieval
Yan, X., Lau, R. Y. K., Song, D., Li, X., & Ma, J. (2011) · ACM Transactions on Information Systems · doi:10.1145/1993036.1993039 · Cited by 56
“The interrater agreement of the document rankings as measured by the Kappa value [Fleiss 1971] for the IT domain (0.8545) is very close to that obtained from the medical domain (0.8501).”
MethodResultInfluentialProbing Internal Assumptions of the Revised Bloom’s Taxonomy
Larsen, T. M., Endo, B. H., Yee, A. T., Do, T., & Lo, S. M. (2022) · CBE—Life Sciences Education · doi:10.1187/cbe.20-08-0170 · Cited by 54
“Fleiss’ kappa, a generalized form of Cohen’s kappa beyond two raters, was used to measure interrater reliability in the subset coded by all three researchers (Fleiss, 1971).”
MethodAssociations between depression, anxious arousal and manifestations of psychological inflexibility
Gilbert, K. E., Tonge, N. A., & Thompson, R. J. (2019) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2018.09.006 · Cited by 53
“Interrater reliability for the presence or absence of current MDD or persistent depressive disorder (Fleiss, 1971) was κ =0.80.”
InfluentialSchiller, P., Burchardi, N., Niestroj, M., & Kieser, M. (2012) · Trials · doi:10.1186/1745-6215-13-214 · Cited by 53
“Fleiss’ Kappa [17].”
Audio Caption: Listen and Tell
Wu, M., Dinkel, H., & Yu, K. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682377 · Cited by 52
Pirrone, F., Ripamonti, A., Garoni, E. C., Stradiotti, S., & Albertini, M. (2017) · Journal of Veterinary Behavior · doi:10.1016/j.jveb.2017.07.004 · Cited by 51
“…of the experimenter who analyzed the videos was 206 computed by coding of independent samples of videotaped sessions twice several weeks apart and 207 calculating the percentage of agreement (Landis and Koch, 1977; Fleiss, 1971), which revealed a 208 kappa value of 0.85 (95% CI: 0.81-0.92).”
MethodInfluentialProoFVer: Natural Logic Theorem Proving for Fact Verification
Krishna, A., Riedel, S., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00503 · Cited by 51
Mekontso Dessap, A., Deux, J.-F., Habibi, A., Abidi, N., Godeau, B., Adnot, S., Brun-Buisson, C., Rahmouni, A., Galacteros, F., & Maitre, B. (2014) · Thorax · doi:10.1136/thoraxjnl-2013-203775 · Cited by 50
Happily Ever After? A Content Analysis of Romantic Ideals in Disney Princess Films
Hefner, V., Firchau, R.-J., Norton, K., & Shevel, G. (2017) · Communication Studies · doi:10.1080/10510974.2017.1365092 · Cited by 50
“This allows for more than two coders and makes corrections for agreements based on chance (Fleiss, 1971; Riffe, Lacy, & Fico, 2005).”
BackgroundEvaluation of the signal quality of wrist-based photoplethysmography
Pradhan, N., Rajan, S., & Adler, A. (2019) · Physiological Measurement · doi:10.1088/1361-6579/ab225a · Cited by 50
“Agreement between raters, Pi, was computed for each PPG segment using (1), the Fleiss (1971) κ.”
MethodCan Neural Machine Translation be Improved with User Feedback?
Kreutzer, J., Khadivi, S., Matusov, E., & Riezler, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers) · doi:10.18653/v1/n18-3012 · Cited by 49
“12 (Fleiss, 1971).”
A Review on Fact Extraction and Verification
Bekoulis, G., Papagiannopoulou, C., & Deligiannis, N. (2023) · ACM Computing Surveys · doi:10.1145/3485127 · Cited by 48
Low, Y. S., Jackson, M. L., Hyde, R. J., Brown, R. E., Sanghavi, N. M., Baldwin, J. D., Pike, C. W., Muralidharan, J., Hui, G., Alexander, N., et al. (2025) · DIGITAL HEALTH · doi:10.1177/20552076251348850 · Cited by 51
Childhood and family factors in the development of health anxiety: A systematic review
Thorgaard, M. V., Frostholm, L., & Rask, C. U. (2018) · Children's Health Care · doi:10.1080/02739615.2017.1318390 · Cited by 46
“Pairwise inter-rater agreement assessed by kappa (K) (Fleiss, 1971; Landis & Koch, 1977) was 0.”
MethodInfluentialMethods for assessing accuracy and reliability in functional MRI
Le, T. H., & Hu, X. P. (1997) · NMR in Biomedicine · doi:10.1002/(sici)1099-1492(199706/08)10:4/5<160::aid-nbm458>3.0.co;2-a · Cited by 49
“Later, Fleiss generalized the statistics kappa to the measurement of agreement among many raters.(20) The formulations given by Fleiss is adapted to fMRI studies with repeated experiments.”
InfluentialWhat are measures of distress tolerance really capturing? A mixed methods analysis.
Ameral, V., Palm Reed, K. M., Cameron, A., & Armstrong, J. L. (2014) · Psychology of Consciousness: Theory, Research, and Practice · doi:10.1037/cns0000024 · Cited by 45
Kodish, S. R., Bio, F., Oemcke, R., Conteh, J., Beauliere, J. M., Pyne-Bailey, S., Rohner, F., Ngnie-Teta, I., Jalloh, M. B., & Wirth, J. P. (2019) · PLOS Neglected Tropical Diseases · doi:10.1371/journal.pntd.0007645 · Cited by 44
Jucaite, A., Cselényi, Z., Kreisl, W. C., Rabiner, E. A., Varrone, A., Carson, R. E., Rinne, J. O., Savage, A., Schou, M., Johnström, P., et al. (2022) · Movement Disorders · doi:10.1002/mds.28814 · Cited by 43
“24,25 Note that visual readers tended to be more “ cautious ” than the machine learning approach in the recognition of cerebellar type MSA (MSA-C) in contrast with parkinsonian type MSA (MSA-P) as indicated by the higher speci fi city and lower sensitivity.”
ResultAnalogy Generation by Prompting Large Language Models: A Case Study of InstructGPT
Bhavya, B., Xiong, J., & Zhai, C. (2022) · Proceedings of the 15th International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.25 · Cited by 40
Calhoun, P. S., Levin, H. F., Dedert, E. A., Johnson, Y., The VA Mid‐Atlantic Mental Illness Research, E., & Beckham, J. C. (2011) · Journal of Traumatic Stress · doi:10.1002/jts.20634 · Cited by 40
“Fleiss’ (1971) kappa was calculated across 17 clinical interviewers for seven SCID training videos with reliability for the PTSD diagnosis of 1.0.”
MethodInfluentialUnveiling the Implicit Toxicity in Large Language Models
Wen, J., Ke, P., Sun, H., Zhang, Z., Li, C., Bai, J., & Huang, M. (2023) · arXiv · doi:10.48550/arxiv.2311.17391 · Cited by 40
“All annotators are college students majoring in English, achieving a moderate to substantial inter-annotator agreement measured by Fleiss’ Kappa (Fleiss, 1971).”
MethodThe Reliability of Clinical Diagnoses: State of the Art
Kraemer, H. C. (2014) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032813-153739 · Cited by 39
“A form of intraclass kappa is available for this situation, and it estimates the same parameter as does the one with M = 2 (Fleiss 1971, 1981).”
MethodIntoxicated Witnesses: Testing the Validity of the Alcohol Myopia Theory
Crossland, D., Kneller, W., & Wilcock, R. (2016) · Applied Cognitive Psychology · doi:10.1002/acp.3209 · Cited by 39
Human-centered neural reasoning for subjective content processing: Hate speech, emotions, and humor
Kazienko, P., Bielaniewicz, J., Gruza, M., Kanclerz, K., Karanowski, K., Miłkowski, P., & Kocoń, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.010 · Cited by 38
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Su, D., Li, X., Zhang, J., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.61 · Cited by 37
Simple but temporally unpredictable puzzles are cognitive enrichment
Krebs, B., & Watters, J. (2017) · Animal Behavior and Cognition · doi:10.12966/abc.09.02.2017 · Cited by 37
“We ensured inter-rater reliabilities by having two observers record data at the same time, then calculating Fleiss’ kappa (Fleiss, 1971) from the simultaneously collected data.”
MethodRead before Generate! Faithful Long Form Question Answering with Machine Reading
Su, D., Li, X., Zhang, J., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · arXiv · doi:10.48550/arxiv.2203.00343 · Cited by 37
Comparing and combining Content‐ and Citation‐based approaches for plagiarism detection
Pertile, S. D. L., Moreira, V. P., & Rosso, P. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23593 · Cited by 36
“The average agreement for the evaluations (not considering the third evaluation, which was used for the disagreement cases) was 84% for ACL and 80% for PubMed, and the Fleiss kappa (Fleiss, 1971) was 0.”
BackgroundMethodInfluentialSalman, I. M., Ameer, O. Z., Khanfar, M. A., & Hsieh, Y.-H. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1495378 · Cited by 36
“For SAQ scores, the rounded scores per difficulty level for each AI tool were compared between evaluators, and a Fleiss’ Kappa test (24) was applied to assess inter-rater reliability among the three evaluators, with values interpreted as follows: Kappa 0 indicates poor agreement, 0–0.20 slight…”
InfluentialMorally Homogeneous Networks and Radicalism
Atari, M., Davani, A. M., Kogon, D., Kennedy, B., Ani Saxena, N., Anderson, I., & Dehghani, M. (2022) · Social Psychological and Personality Science · doi:10.1177/19485506211059329 · Cited by 36
“agreement coefficients for multiple annotators (Fleiss, 1971), after correction for the low prevalence of labels, prevalence- and bias-adjusted kappa (PABAK; Byrt et al.”
BackgroundMethodEnhancing evidence-based medicine with natural language argumentative analysis of clinical trials.
Mayer, T., Marro, S., Cabrio, E., & Villata, S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102098 · Cited by 36
“As the statistical measure for assessing the reliability of the annotations, we used Fleiss’ kappa [54], a generalization of Scott’s pi.”
MethodEvaluating training methods for facial image comparison: The face shape strategy does not work
Towler, A., White, D., & Kemp, R. I. (2014) · Perception · doi:10.1068/p7676 · Cited by 35
“Further, analysis of interrater reliability using Fleiss’s l (see Fleiss, 1971) also revealed very poor agreement between participants (l = 0.10).”
MethodInfluentialGenerating Literal and Implied Subquestions to Fact-check Complex Claims
Chen, J., Sriram, A., Choi, E., & Durrett, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.229 · Cited by 34
Generating Literal and Implied Subquestions to Fact-check Complex Claims
Chen, J., Sriram, A., Choi, E., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2205.06938 · Cited by 34
“The annotations agree with a Fleiss’ kappa (Fleiss, 1971) value of 0.”
MethodA Critical Evaluation of Evaluations for Long-form Question Answering
Xu, F., Song, Y., Iyyer, M., & Choi, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.181 · Cited by 33
Husbandry and enclosure influences on penguin behavior and conservation breeding
Marshall, A. R., Deere, N. J., Little, H. A., Snipp, R., Goulder, J., & Mayer-Clarke, S. (2016) · Zoo Biology · doi:10.1002/zoo.21313 · Cited by 33
Asymptotic variability of (multilevel) multirater kappa coefficients
Vanbelle, S. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218794733 · Cited by 33
Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
Shah, C., White, R., Andersen, R., Buscher, G., Counts, S., Das, S., Montazer, A., Manivannan, S., Neville, J., Rangan, N., et al. (2025) · ACM Transactions on the Web · doi:10.1145/3732294 · Cited by 33
“Write an introduction about geothermal energy; Modify a poem into different formats; Improve a report and find adverbs and connectors.”
Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
Shah, C., White, R. W., Andersen, R., Buscher, G., Counts, S., Das, S. S. S., Montazer, A., Manivannan, S., Neville, J., Ni, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.13063 · Cited by 33
DuBois, L. Z., Macapagal, K. R., Rivera, Z., Prescott, T. L., Ybarra, M. L., & Mustanski, B. (2015) · Archives of Sexual Behavior · doi:10.1007/s10508-015-0521-5 · Cited by 32
“75 was observedacrosscoders, indicatingagood(Dedoose,2012;Landis & Koch, 1977)-to-excellent level of reliability (Fleiss, 1971).”
Ibnidris, A., Robinson, J. N., Stubbs, M., Piumatti, G., Govia, I., & Albanese, E. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02018-y · Cited by 30
“After the completion of the screening phase, the inter-rater reliability and agreement between the three reviewers were measured using Fleiss kappa [28] and agreement percentage in RStudio [29].”
MethodSpecifying Formative Constructs in Information Systems Research1
Petter, S., Straub, D., & Rai, A. (2007) · MIS Quarterly · doi:10.2307/25148814 · Cited by 32
Unsupervised tip-mining from customer reviews
Zhu, D., Lappas, T., & Zhang, J. (2018) · Decision Support Systems · doi:10.1016/j.dss.2018.01.011 · Cited by 32
Morjaria, L., Burns, L., Bracken, K., Ngo, Q. N., Lee, M., Levinson, A. J., Smith, J., Thompson, P., & Sibbald, M. (2023) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205231204178 · Cited by 31
A Simple Guide to Inter-rater, Intra-rater and Test-retest Reliability for Animal Behaviour Studies
Harvey, N. D. (2021) · doi:10.31219/osf.io/8stpy · Cited by 31
“…3 or more raters, such as Krippendorf’s alpha (Hayes & Krippendorff, 2007) (which can be used with less restrictions than Cohen’s kappa), Fleiss’ Kappa (Fleiss, 1971) (minor differences from Krippendorff’s alpha), and Gwet’s AC 1 (Gwet, 2008) (described as more paradox resistant than alternatives).”
MethodEffects of GenAI Interventions on Student Academic Performance: A Meta-Analysis
Gu, J., & Yan, Z. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251349620 · Cited by 30
“Inter-rater reliability (kappa) between the two coders was 0.48, indicating a moderate agreement (Fleiss, 1971).”
InfluentialLooking for a handsome carpenter! Debiasing GPT-3 job advertisements
Borchers, C., Gala, D., Gilburt, B., Oravkin, E., Bounsi, W., Asano, Y. M., & Kirk, H. (2022) · Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.22 · Cited by 29
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.121 · Cited by 29
Interrater agreement of nasal endoscopy in patients with a prior history of endoscopic sinus surgery
McCoul, E. D., Smith, T. L., Mace, J. C., Anand, V. K., Senior, B. A., Hwang, P. H., Stankiewicz, J. A., & Tabaee, A. (2012) · International Forum of Allergy & Rhinology · doi:10.1002/alr.21058 · Cited by 28
Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
Hua, A., Tang, K., Gu, C., Gu, J., Wong, E., & Qin, Y. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1006 · Cited by 28
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation
Feng, J., Sun, Q., Xu, C., Zhao, P., Yang, Y., Tao, C., Zhao, D., & Lin, Q. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.405 · Cited by 26
Perception of emotional expressions in different representations using facial feature points
Afzal, S., Sezgin, T. M., Gao, Y., & Robinson, P. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349549 · Cited by 25
“Since the experiment involves multiple raters rating multiple categories, we compute Fleiss’ kappa as a measure of inter-rater reliability or agreement [ 8 ].”
MethodInference of development activities from interaction with uninstrumented applications
Bao, L., Xing, Z., Xia, X., Lo, D., & Hassan, A. E. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9547-8 · Cited by 25
“We also compute the Kappa statistic (Fleiss 1971), which is a measure of agreement between the predictions and the actual labels.”
MethodDean, B. J. F., Jones, L. D., Palmer, A. J. R., Macnair, R. D., Brewer, P. E., Jayadev, C., Wheelton, A. N., Ball, D. E. J., Nandra, R. S., Aujla, R. S., et al. (2016) · Bone & Joint Research · doi:10.1302/2046-3758.55.2000596 · Cited by 24
Perceptions of Creativity in Software Engineering Research and Practice
Mohanani, R., Ram, P., Lasisi, A., Ralph, P., & Turhan, B. (2017) · 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2017.21 · Cited by 24
“7 denoted a medium-high agreement [21].”
Hurdles to Artificial Intelligence Deployment: Noise in Schemas and “Gold” Labels
Abdalla, M., & Fine, B. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220056 · Cited by 24
“Like Cohen κ , it accounts for chance agreement between all raters (13).”
BackgroundUICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset
Duan, P., Cheng, C.-Y., Li, G., Hartmann, B., & Li, Y. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676381 · Cited by 24
“The three annotators had a Fleiss’ Kappa [11] inter-rater agreement score of 0.30, which indicates fair agreement.”
BackgroundMethodInfluentialQuality assessment of meta-analyses of RCTs of pharmacotherapy in major depressive disorder.
Hemels, M. E. H., Vicente, C., Sadri, H., Masson, M. J., & Einarson, T. R. (2004) · Current Medical Research and Opinion · doi:10.1185/030079904125003197 · Cited by 23
“68 EXCLUDED ARTICLES: - Inappropriate disease/comorbidity (9) - Inappropriate drug/comparator (13) - Inappropriate outcome measure (15) - Not retrievable (5) - Inappropriate patient population (15) - Inappropriate study design/not original research (11)”
BackgroundMultiple‐rater kappas for binary data: Models and interpretation
Stoyan, D., Pommerening, A., Hummel, M., & Kopp‐Schneider, A. (2018) · Biometrical Journal · doi:10.1002/bimj.201600267 · Cited by 23
“Fleiss' 𝜅 is a popular statistical measure for assessing the agreement of 𝑟 raters, where 𝑟 > 2, which assigns categorical ratings to a number of items (Fleiss, 1971; Fleiss, Levin & Paik, 2003).”
BackgroundMethodEco-Feedback for Food Waste Reduction in a Student Residence
Lim, V., Bartram, L., Funk, M., & Rauterberg, M. (2021) · Frontiers in Sustainable Food Systems · doi:10.3389/fsufs.2021.658898 · Cited by 22
“In other words, Fleiss’ kappameasures the level of agreement that a certain item or subject belongs to a certain category (Fleiss, 1971).”
BackgroundKuo, P. H., Yoo, D. C., Avery, R., Seltzer, M., Calais, J., Nagarajah, J., Weber, W. A., Fendler, W. P., Hofman, M. S., Krause, B. J., et al. (2023) · Journal of Nuclear Medicine · doi:10.2967/jnumed.122.265077 · Cited by 22
On winnowing: The impact of scarcity on allocators' evaluations of candidates for a resource
Ross, M., & Ellard, J. H. (1986) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(86)90021-1 · Cited by 21
“We calculated agreement using Kappa (K), a statistic that shows the proportion of agreement remaining after chance agreement is removed (Fleiss, 1971; Watkins, 1979).”
MethodExploring Differences in Measurement and Reporting of Classroom Observation Inter-Rater Reliability
Wilhelm, A. G., Rouse, A. G., & Jones, F. (2018) · University of Massachusetts Amherst · doi:10.7275/at67-md25 · Cited by 20
Krippendorff, K. (2016) · Methodology · doi:10.1027/1614-2241/a000119 · Cited by 22
““argued that the error model of (Cohen’s, 1960) κ, (Scott’s, 1955; Fleiss, 1971) π, and (Krippendorff’s, 2013) α is implausible as they assume that coders work by flipping coins all the time.”
BackgroundChittaro, L., & Vianello, A. (2016) · Interacting with Computers · doi:10.1093/iwc/iwv044 · Cited by 19
Referencing practices in physical geography: How well do we cite what we write?
Haussmann, N. S., McIntyre, T., Bumby, A. J., & Loubser, M. J. (2013) · Progress in Physical Geography: Earth and Environment · doi:10.1177/0309133313482135 · Cited by 20
“We also tested the overall agreement between all four raters using an Exact Fleiss’ Kappa statistic (Conger, 1980; Fleiss, 1971), and agreement was assessed according to the classes proposed by Viera and Garrett (2005).”
MethodDalal, S., Tilwani, D., Gaur, M., Jain, S., Shalin, V. L., & Sheth, A. P. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483577 · Cited by 19
“MHPs attained an inter-annotator agreement of 0.85 in PRIMATE and 0.77 in CLEF e-Risk using the Fleiss Kappa reliability measurement scale [52].”
MethodInfluentialAgreement between clinical and MINI diagnoses in outpatients with mood and anxiety disorders
Verhoeven, F. E. A., Swaab, L. S. M. A., Carlier, I. V. E., van Hemert, A. M., Zitman, F. G., Ruhé, H. G., Schoevers, R. A., & Giltay, E. J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.06.041 · Cited by 19
Facebook Use of Sensitive Data for Advertising in Europe
Cabañas, J. G., Cuevas, Á., & Cuevas, R. (2018) · arXiv · doi:10.48550/arxiv.1802.05030 · Cited by 16
“We have evaluated the voters’ agreement across the sensitive categories assigned to ad preferences labeled as sensitive using the Fleiss’ Kappa test [10][11].”
MethodHubert's multi‐rater kappa revisited
Martín Andrés, A., & Álvarez Hernández, M. (2020) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12167 · Cited by 16
A new coefficient of interrater agreement: The challenge of highly unequal category proportions.
van Oest, R. (2019) · Psychological Methods · doi:10.1037/met0000183 · Cited by 14
“If R = 2, (1) reduces to the fraction of items on which both raters agree (Fleiss, 1971).”
InfluentialIdentifying Annotator Bias: A new IRT-based method for bias identification
Amidei, J., Piwek, P., & Willis, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.421 · Cited by 14
“(1)Some examples of the Kappa statistic are Cohen’s κ (Cohen, 1960), Fleiss’ κ (Fleiss, 1971) and Krippendorff’s α (Krippendorff, 1980).”
Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL
Cheng, N., Yan, Z., Wang, Z., Li, Z., Yu, J., Zheng, Z., Tu, K., Xu, J., & Han, W. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-981-97-5663-6_5 · Cited by 13
Towards Geographic Inclusion in the Evaluation of Text-to-Image Models
Hall, M., Bell, S. J., Ross, C., Williams, A., Drozdzal, M., & Soriano, A. R. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658927 · Cited by 11
“We binarize the presence of each code per response and evaluate inter-coder agreement using Fleiss’ Kappa [16].”
MethodRating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
Haldar, R., & Hockenmaier, J. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1361 · Cited by 11
“Developed by Klaus Krip-pendorff in the context of content analysis, α generalizes many classical agreement statistics (such as Cohen’s (Cohen, 1960) or Fleiss’ (Fleiss, 1971) Kappa) and applies to any number of coders, any number of categories or scale values, and any measurement level (nominal, ordinal, interval, ra…”
InfluentialBehavioural synchrony in two species of communally housed captive penguins
Foerder, P., Chodorow, M., & Moore, D. E. (2013) · Behaviour · doi:10.1163/1568539x-00003100 · Cited by 9
“Our analysis of the data compares intra- and inter-species synchrony using two measures, Kappa (Fleiss, 1971; Siegel & Castellan, 1988; Rook & Penning, 1991) and cross-correlation (Bjørnstad et al., 1999).”
MethodDeciphering Stereotypes in Pre-Trained Language Models
Ma, W., Scheible, H., Wang, B., Veeramachaneni, G., Chowdhary, P., Sun, A., Koulogeorge, A., Wang, L., Yang, D., & Vosoughi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.697 · Cited by 8
“Inter-annotator agreement for these validation tasks was quantified using Fleiss’ κ (Fleiss, 1971), with scores exceeding 0.74 for both sets of validations.”
BackgroundMethodInfluentialMeasuring Agreement Using Guessing Models and Knowledge Coefficients
Moss, J. (2023) · Psychometrika · doi:10.1007/s11336-023-09919-4 · Cited by 8
Bazoge, A., Wargny, M., Constant dit Beaufils, P., Morin, E., Daille, B., Gourraud, P.-A., & Hadjadj, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110609 · Cited by 6
UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images
Qu, Y., Shen, X., Wu, Y., Backes, M., Zannettou, S., & Zhang, Y. (2025) · Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3719027.3765088 · Cited by 6
“In this study, we use the gpt-4-vision-preview checkpoint.”
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.