Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 901–1,000 of 1,118
A measure of psychotherapeutic competence derived from cognitive analytic therapy
Bennett, D., & Parry, G. (2004) · Psychotherapy Research · doi:10.1093/ptr/kph016 · Cited by 77
“Cohen’s kappa (Fleiss, 1971) was used to calculate interrater reliability for the CCAT domain scores because this takes into account chance agreement.”
MethodMeasures of agreement between many raters for ordinal classifications
Nelson, K. P., & Edwards, D. (2015) · Statistics in Medicine · doi:10.1002/sim.6546 · Cited by 84
“Existing summary measures for assessing agreement between multiple raters when ordinal classifications are being examined include Fleiss’ kappa [9], Light’s and Conger’s kappa [10, 11], Kraemer’s kappa coefficient [12], and an AC2 statistic [13].”
Extensive Leukoaraiosis Is Associated With High Early Risk of Recurrence After Ischemic Stroke
Kim, G., Park, K., Avery, R., Helenius, J., Rost, N. S., Rosand, J. M., Rosen, B. R., & Ay, H. (2013) · Stroke · doi:10.1161/strokeaha.113.003004 · Cited by 80
“) in a set of 100 consecutive patients using κ statistics.(7) The κ coefficient for PLA and SLA were 0.”
BackgroundRecovering Concept Prerequisite Relations from University Course Dependencies
Liang, C., Ye, J., Wu, Z., Pursel, B., & Giles, C. L. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10550 · Cited by 104
“Based on our setting of the labeling process, Fleiss’ kappa κ (Fleiss 1971) is used to assess the reliability of agreement between annotators.”
MethodThe Validity of Male Patients' Self-Reports Regarding Prostate Cancer Screening
Jordan, T. R., Price, J. H., King, K. A., Masyk, T., & Bedell, A. W. (1999) · Preventive Medicine · doi:10.1006/pmed.1998.0430 · Cited by 77
Learning classification models with soft-label information
Nguyen, Q., Valizadegan, H., & Hauskrecht, M. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001964 · Cited by 99
Empowerment Praxis: Community Organizing to Redress Systemic Health Disparities
Douglas, J. A., Grills, C. T., Villanueva, S., & Subica, A. M. (2016) · American Journal of Community Psychology · doi:10.1002/ajcp.12101 · Cited by 81
An exploration of crowdsourcing citation screening for systematic reviews
Mortensen, M. L., Adam, G. P., Trikalinos, T. A., Kraska, T., & Wallace, B. C. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1252 · Cited by 76
“We report Fleiss Kappa scores for each review (calculated independently for each question) in Table 5.(22) One can observe quite a bit of variance; agreement ranges from poor to moderate.”
MethodSpinde, T., Rudnitckaia, L., Mitrović, J., Hamborg, F., Granitzer, M., Gipp, B., & Donnay, K. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102505 · Cited by 93
“Based on the way in which we have organized our crowdsourcing workflow, i.e. having 10 annotators per task, we have decided to use Fleiss (1971) to assess the inter-annotator agreement.”
Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models
Qu, Y., Shen, X., He, X., Backes, M., Zannettou, S., & Zhang, Y. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3616679 · Cited by 274
“These unsafe images in the training data fundamentally explain why models generate unsafe images.”
BackgroundComplex Sequencing Rules of Birdsong Can be Explained by Simple Hidden Markov Processes
Katahira, K., Suzuki, K., Okanoya, K., & Okada, M. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024516 · Cited by 76
“To evaluate the agreement between manual annotations by different human annotators and between manual and model annotation, we used Fleiss’s k coefficient and Cohen’s k coefficient, respectively [12].”
MethodReliability and accuracy of skeletal muscle imaging in limb-girdle muscular dystrophies
ten Dam, L., van der Kooi, A. J., van Wattingen, M., de Haan, R. J., & de Visser, M. (2012) · Neurology · doi:10.1212/wnl.0b013e31826e9b73 · Cited by 76
Mwinzi, P. N. M., Kittur, N. D., Ochola, E. A., Cooper, P. J., Campbell, C. H., King, C. H., & Colley, D. G. (2015) · Frontiers in Public Health · doi:10.3389/fpubh.2015.00048 · Cited by 76
“Fleiss kappa statistical analysis (25) FIGURE 1 | Flow diagram of post-praziquantel treatment study, Kenya.”
MethodDNA Barcode Sequence Identification Incorporating Taxonomic Hierarchy and within Taxon Variability
Little, D. P. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020552 · Cited by 89
“Similarity among SIDEs was quantified using Fleiss'”
Is Arson an Aggressive Act or a Property Offence? A Controlled Study of Psychiatric Referrals
Hill, R. W., Langevin, R., Paitich, D., Handy, L., Russon, A., & Wilkinson, L. (1982) · The Canadian Journal of Psychiatry · doi:10.1177/070674378202700809 · Cited by 76
“The two raters were compared on each item by Kappa (9).”
Methodde Albuquerque, J. P., Herfort, B., & Eckle, M. (2016) · Remote Sensing · doi:10.3390/rs8100859 · Cited by 77
Assessing the preventability of emergency hospital admissions
Bigby, J., Dunn, J. J., Goldman, L., Adams, J., Jen, P., Landefeld, C. S., & Komaroff, A. L. (1987) · The American Journal of Medicine · doi:10.1016/0002-9343(87)90938-7 · Cited by 76
A Generalized Kappa Coefficient
Uebersax, J. S. (1982) · Educational and Psychological Measurement · doi:10.1177/0013164482421018 · Cited by 76
A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges
Janson, H., & Olsson, U. (2004) · Educational and Psychological Measurement · doi:10.1177/0013164403260195 · Cited by 81
“Fleiss (1971) and Landis and Koch (1977) used one-way ANOVA designs to obtain agreement estimates for univariate categorical data.”
MethodThe Toe Walking Tool: A novel method for assessing idiopathic toe walking children
Williams, C., Tinley, P., & Curtin, M. (2010) · Gait & Posture · doi:10.1016/j.gaitpost.2010.07.011 · Cited by 84
Gender Roles and Stereotypes about Science Careers Help Explain Women and Men’s Science Pursuits
Stout, J. G., Grunberg, V. A., & Ito, T. A. (2016) · Sex Roles · doi:10.1007/s11199-016-0647-5 · Cited by 106
“Courses within each category were coded by three independent coders (κ = .83; Fleiss 1971); disagreements about category coding were resolved by discussion until consensus was reached (e.g., reading descriptions of majors and required courses for those majors on the university website; consulting…”
MethodKim, H., Coutrix, C., & Roudaut, A. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174193 · Cited by 75
“This work resulted in an average Pearson correlation between raters [8,9] of 0.319, indicating the annotation subjective .”
BackgroundInfluentialAppropriateness of Domiciliary Oxygen Delivery
Guyatt, G. H., Mckim, D. A., Austin, P., BRYAN, R. E. J., Norgren, J., Weaver, B., & Goldstein, R. S. (2000) · CHEST Journal · doi:10.1378/chest.118.5.1303 · Cited by 75
Modelling patterns of agreement for nominal scales
Roberts, C. (2007) · Statistics in Medicine · doi:10.1002/sim.2945 · Cited by 84
“Fleiss [2] described a method for assessing the reliability of nominal scales using a summary kappa coefficient for multiple raters.”
MethodDemystifying a Q-Matrix for Making Diagnostic Inferences About L2 Reading Skills
Jang, E. E. (2009) · Language Assessment Quarterly · doi:10.1080/15434300903071817 · Cited by 75
“Inter-rater reliability was assessed using Fleiss' Kappa statistic ( Fleiss, 1971 ) for categorical ratings from more than two raters.”
MethodWeighted Least-Squares Approach for Comparing Correlated Kappa
Barnhart, H. X., & Williamson, J. M. (2002) · Biometrics · doi:10.1111/j.0006-341x.2002.01012.x · Cited by 80
“Both Cohen’s kappa and weighted kappa (Cohen, 1968; Fleiss, 1971) can be used to assess the interrater agreement.”
BackgroundMethodA time series interaction analysis method for building predictive models of learners using log data
Brooks, C. A., Thompson, C. D. S., & Teasley, S. D. (2015) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2723576.2723581 · Cited by 97
“…to all three research questions we report the number of correct classifications, the number of incorrect classifications, and Fleiss’ κ (kappa) [9] as a measure of agreement between observed data and predicted data. κ is chance corrected, and ranges from -1 to 1 where 0 is an indication of…”
BackgroundStatus and Corporate Illegality: Illegal Loan Recovery Practices of Commercial Banks in India
Krishnan, R., & Kozhikode, R. K. (2014) · Academy of Management Journal · doi:10.5465/amj.2012.0508 · Cited by 88
Linking the Thoughts: Analysis of Argumentation Structures in Scientific Publications
Kirschner, C., Eckle‐Kohler, J., & Gurevych, I. (2015) · Workshop on Argument Mining (ArgMining) · doi:10.3115/v1/w15-0501 · Cited by 125
“Multi-π is the extension of Scott’s π by Fleiss (1971).”
Survival Analysis of Implant in Patients With Diabetes Mellitus
Annibali, S., Pranno, N., Cristalli, M. P., Monaca, G. L., & Polimeni, A. (2016) · Implant Dentistry · doi:10.1097/id.0000000000000478 · Cited by 75
“of heterogeneity was not performed.(10) The following potential confounding factors were used to evaluate heterogeneity between the studies:”
MethodBehavior Problems in Children of Parents with Anxiety Disorders
Silverman, W. K., Cerny, J. A., Nelles, W. B., & Burke, A. E. (1988) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-198811000-00020 · Cited by 85
Brinsfield, C. T. (2009) · OhioLink ETD Center (Ohio Library and Information Network) · Cited by 171
Agreement Analysis: What He Said, She Said Versus You Said
Vetter, T. R., & Schober, P. R. (2018) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002924 · Cited by 80
Incorporation of a global perspective into data-driven analysis of maritime collision accident risk
Li, H., Çelik, C., Bashir, M. B., Zou, L., & Yang, Z. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110187 · Cited by 85
Model-Based Classification Methods of Global Patterns in Dermoscopic Images
Sáez, A., Serrano, C., & Acha, B. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2305769 · Cited by 82
A neurological examination score for the assessment of spinocerebellar ataxia 3 (SCA3)
Kieling, C., Rieder, C. R. D. M., Silva, A. C. F., Saute, J. A. M., Cecchin, C. R., Monte, T. L., & Jardim, L. B. (2008) · European Journal of Neurology · doi:10.1111/j.1468-1331.2008.02078.x · Cited by 73
“Interrater reliability was expressed using the kappa statistic [14].”
MethodAutomatic detection of patient-ventilator asynchrony by spectral analysis of airway flow
Gutiérrez, G., Ballarino, G. J., Türkan, H., Abril, J., Cruz, L. D. L., Edsall, C. W., George, B., Gutierrez, S., Jha, V. M., & Ahari, J. (2011) · Critical Care · doi:10.1186/cc10309 · Cited by 74
“We conducted a Cohen Kappa test modified by Fleiss for three independent observers [15] and found a statistic = 0.”
MethodModeling kappa for measuring dependent categorical agreement data
Williamson, J. M., Lipsitz, S. R., & Manatunga, A. K. (2000) · Biostatistics · doi:10.1093/biostatistics/1.2.191 · Cited by 74
“Historically, κ has been used to determine the agreement of binary (Cohen, 1960) and categorical (Fleiss, 1971) outcomes between raters.”
BackgroundMethodValue of chest radiography in phenotyping chronic obstructive pulmonary disease
Miniati, M., Monti, S., Stolk, J. N., Mirarchi, G., Falaschi, F., Rabinovich, R., Canapini, C., Roca, J., & Rabe, K. F. (2007) · European Respiratory Journal · doi:10.1183/09031936.00095607 · Cited by 74
Testing the Homogeneity of Kappa Statistics
Donner, A., Eliasziw, M., & Klar, N. (1996) · Biometrics · doi:10.2307/2533154 · Cited by 74
“As noted by Hale and Fleiss (1993), this model also permits measurement of agreement in several other settings than that considered here: the reliability of a single rater based on independent replicate assignments by that rater; the reproducibility of evaluations made by members of a pool of raters when different pai…”
MethodLax Vox as a Voice Training Program for Teachers: A Pilot Study
Mailänder, E., Mühre, L., & Latoszek, B. B. V. (2016) · Journal of Voice · doi:10.1016/j.jvoice.2016.04.011 · Cited by 84
Stigma and diurnal cortisol among transitioning transgender men
DuBois, L. Z., Powers, S. I., Everett, B. G., & Juster, R. (2017) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2017.05.008 · Cited by 90
The risk of bias in systematic reviews tool showed fair reliability and good construct validity
Bühn, S., Mathes, T., Prengel, P., Wegewitz, U. E., Ostermann, T., Robens, S., & Pieper, D. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.06.019 · Cited by 82
Safety and Security Co-Analyses: A Systematic Literature Review
Lisova, E., Šljivo, I., & Čaušević, A. (2018) · IEEE Systems Journal · doi:10.1109/jsyst.2018.2881017 · Cited by 74
Identifying Accident-Prone Locations Using Fuzzy Pattern Recognition
Sayed, T., Abdelwahab, W., & Navin, F. (1995) · Journal of Transportation Engineering · doi:10.1061/(asce)0733-947x(1995)121:4(352 · Cited by 74
Wang, G., Zuluaga, M. A., Pratt, R., Aertsen, M., Doel, T., Klusmann, M., David, A. L., Deprest, J., Vercauteren, T., & Ourselin, S. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.04.009 · Cited by 73
“The agreement between different segmentations was measured by Fleiss’ kappa coefficient ( Fleiss, 1971 ): where ¯ P a is the relative observed agreement, and ¯ P e is the hypothetical probability of chance agreement.”
MethodScience models as value-added services for scholarly information systems
Mutschke, P., Mayr, P., Schaer, P., & Sure, Y. (2011) · Scientometrics · doi:10.1007/s11192-011-0430-x · Cited by 87
“Liu et al (2005) discovered a strong correlation between program committee membership and betweenness in co-authorship networks. Mutschke and Quan-Haase (2001) observed a high correlation of betweenness in coauthorship networks and betweenness of the author„s topics in keyword networks.”
BackgroundSources of Distress in First-Episode Psychosis: A Systematic Review and Qualitative Metasynthesis
Griffiths, R., Mansell, W., Edge, D., & Tai, S. (2018) · Qualitative Health Research · doi:10.1177/1049732318790544 · Cited by 84
Huang, Q., Xia, X., Lo, D., & Murphy, G. C. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876340 · Cited by 78
“We use Fleiss Kappa [57] to measure the agreement between the two labelers.”
MethodProgress update from the hippocampal subfields group
Olsen, R. K., Carr, V. A., Daugherty, A. M., Joie, R. L., Amaral, R. S. C., Amunts, K., Augustinack, J. C., Bakker, A., Bender, A. R., Berron, D., et al. (2019) · Alzheimer s & Dementia Diagnosis Assessment & Disease Monitoring · doi:10.1016/j.dadm.2019.04.001 · Cited by 90
Beyer, S., Macho, C., Penta, M. D., & Pinzger, M. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09758-x · Cited by 93
“We calculated the Fleiss-Kappa inter-rater agreement (Fleiss 1971) and obtained a κ = 0.49, meaning moderate agreement.”
MethodInfluentialWNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets
Nguyen, D. Q., Vu, T., Rahimi, A., Dao, M. H., Nguyen, L. T., & Doan, L. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.41 · Cited by 83
Weber, E., Krehl, E., & Büttgen, M. (2022) · Journal of Leadership Studies · doi:10.1002/jls.21810 · Cited by 144
Detection of ‘EEG bursts’ in the early preterm EEG: Visual vs. automated detection
Palmu, K., Wikström, S., Hippeläinen, E. T., Boylan, G. B., Hellström-Westas, L. K., & Vanhatalo, S. (2010) · Clinical Neurophysiology · doi:10.1016/j.clinph.2010.02.010 · Cited by 75
High-Resolution CT in Silicosis
Antao, V. C., Pinheiro, G. A., Filho, M. T., Kavakama, J., & ller, N. L. M. (2005) · Journal of Computer Assisted Tomography · doi:10.1097/01.rct.0000160424.56261.bc · Cited by 73
“Interreader agreement in the reading of radiographs and HRCT was assessed using Kappa statistics.(27) A P value ,0.”
MethodDevelopment of a Road Safety Risk Index
de Leur, P., & Sayed, T. (2002) · Transportation Research Record Journal of the Transportation Research Board · doi:10.3141/1784-05 · Cited by 73
Foroohar, A., Tosti, R. J., Richmond, J. M., Gaughan, J. P., & Ilyas, A. M. (2011) · Journal of Orthopaedic Surgery and Research · doi:10.1186/1749-799x-6-38 · Cited by 73
“Inter-observer agreement was assessed via computer-calculated kappa statistics based on the works of Cohen and Fleiss [26,27].”
MethodLo, L.-C., Chen, Y., Chen, W.-J., Cheng, T.-L., & Chiang, J. Y. (2012) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2012/505063 · Cited by 73
“The inter-agreement among TCM doctors is indicated with Fleiss’ kappa coefficient [19], which is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items or classifying items.”
Erba, G., Giussani, G., Juersivich, A., Magaudda, A., Chiesa, V., Laganà, A., Rosa, G. D., Bianchi, E., Langfitt, J. T., & Beghi, E. (2016) · Epilepsia · doi:10.1111/epi.13351 · Cited by 73
Between-Subjects Elicitation Studies: Formalization and Tool Support
Vatavu, R.-D., & Wobbrock, J. O. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858228 · Cited by 78
“Inter-rater reliability has been evaluated using the percent agreement [18,32], Scott’s π statistic [40], and Cohen’s κ (kappa) [8] and Fleiss’ κ [14] coefficients.”
MethodSun, S., Hoyt, W. T., Brockberg, D., Lam, J., & Tiwari, D. (2016) · Journal of Counseling Psychology · doi:10.1037/cou0000172 · Cited by 72
“We used Fleiss’s kappa (Fleiss, 1971), an adaption of Cohen’s kappa for more than two raters, to calculate interrater reliability.”
MethodMining personality traits from social messages for game recommender systems
Yang, H.-C., & Huang, Z.-R. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.11.025 · Cited by 76
Generalization of Scott's Index of Intercoder Agreement
Craig, R. T. (1981) · Public Opinion Quarterly · doi:10.1086/268657 · Cited by 73
Interobserver agreement in assessment of ocular signs in coma.
van den Berge, J. H., Schouten, H. J. A., Boomstra, S., Littel, S. V. D., & Braakman, R. (1979) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.42.12.1163 · Cited by 73
“The null hypothesis that the physicians only agreed by chance was tested by means of a standard normal value Z (Fleiss, 1971).”
BackgroundMethodClassification of discourse functions of affirmative words in spoken dialogue
Gravano, A., Beňuš, Š., Hirschberg, J., Mitchell, S., & Vovsha, I. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-450 · Cited by 74
“Inter-labeler reliability was measured by Fleiss' κ ([16]) at 0.69, where values between 0.6 and 0.8 correspond to substantial agreement.”
MethodInfluentialTyagi, M., Kumar, P., & Kumar, D. (2018) · International Journal of Logistics Research and Applications · doi:10.1080/13675567.2017.1422707 · Cited by 73
“After that, Fleiss (1971, 1981) discussed and explained the consistency statistics in facet, which was applicable to the case of more than two experts.”
MethodMining customer product reviews for product development: A summarization process
Hou, T., Yannou, B., Leroy, Y., & Poirson, É. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.04.069 · Cited by 77
“Fleiss’s kappa (Fleiss, 1971) is widely used to calculate the inter-agreement.”
MethodResultInfluentialSelf-Attention Guided Copy Mechanism for Abstractive Summarization
Xu, S., Li, H., Yuan, P., Wu, Y., He, X., & Zhou, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.125 · Cited by 98
“We evaluate the agreement by Fleiss’ kappa (Fleiss, 1971).”
MethodPredicting the objective and priority of issue reports in software repositories
Izadi, M., Akbari, K., & Heydarnoori, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10085-3 · Cited by 94
“Two variations of kappa for multi-raters (more than two) are Fleiss’ fixed-marginal multi-rater kappa [20] and Randolph’s free-marginal multi-rater kappa [47].”
ScienceQA: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., & Bhattacharyya, P. (2022) · International Journal on Digital Libraries · doi:10.1007/s00799-022-00329-y · Cited by 260
“We compute the inter-annotator agreement ratio in terms of the kappa coefficient [18].”
MethodDoes perception mean learning? Insights from an online peer feedback setting
Noroozi, O., Alqassab, M., Taghizadeh Kerman, N., Banihashem, S. K., & Panadero, E. (2025) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2024.2345669 · Cited by 72
“To determine the inter-rater reliability among coders, we employed Fleiss’ Kappa statistic (Fleiss 1971).”
Testing for publication bias in diagnostic meta‐analysis: a simulation study
Bürkner, P., & Doebler, P. (2014) · Statistics in Medicine · doi:10.1002/sim.6177 · Cited by 73
“Although it can be applied in diagnostic contexts [32, 33], K is generally known as a measure of the agreement between two or (if generalized) more raters [34], but not as a measure of diagnostic accuracy.”
BackgroundHowell, P., Au‐Yeung, J., & Sackin, S. (2000) · Journal of Fluency Disorders · doi:10.1016/s0094-730x(99)00025-x · Cited by 72
“The agreement between the two judges on content/function word was 98% giving a kappa coefficient of 0.96 which is much higher than chance ( Fleiss, 1971 ).”
BackgroundInfluentialKoski, J. M., Saarakkala, S. S., Helle, M. J., Hakulinen, U., Heikkinen, J. O., Hermunen, H., Bálint, P., Bruyn, G. A. W., Filippucci, E., Grassi, W., et al. (2006) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2005.051250 · Cited by 72
Reliability of nutritional screening and assessment tools
Jones, J. M. (2004) · Nutrition · doi:10.1016/j.nut.2003.11.012 · Cited by 84
Johnson, K. E., Neta, G. I., Dember, L. M., Coronado, G. D., Suls, J. M., Chambers, D., Rundell, S. D., Smith, D. H., Liu, B., Taplin, S. H., et al. (2016) · Trials · doi:10.1186/s13063-016-1158-y · Cited by 75
“Interrater agreement or reliability was calculated for each trial using Gwet’s AC1 statistic [7], a more robust version [8] of Fleiss’ Kappa [9].”
MethodParting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks
Kairam, S., & Heer, J. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2820016 · Cited by 72
“We calculate inter-annotator agreement throughout using Fleiss’ κ [14].”
MethodAgreement between Alternative Classifications of Acute Respiratory Distress Syndrome
MEADE, M. O., GUYATT, G. H., COOK, R. J., GROLL, R., KACHURA, J. R., WIGG, M., COOK, D. J., SLUTSKY, A. S., & STEWART, T. E. (2001) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.163.2.2006067 · Cited by 72
Linear Discriminant Function analysis in Neuropsychological Research: Some Uses and abuses
Fletcher, J. M., Rice, W. J., & Ray, R. M. (1978) · Cortex · doi:10.1016/s0010-9452(78)80031-8 · Cited by 72
NERD: evaluating named entity recognition tools in the web of data
Rizzo, G. F., & Raphaël Troncy, A. (2011) · PORTO@Iris · Cited by 71
Drennan, V. M., Halter, M., Brearley, S. A., Carneiro, W., Gabe, J., Gage, H., Grant, R., Joly, L., & de Lusignan, S. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02160 · Cited by 89
Functional connectivity of reflective and brooding rumination in depressed and healthy women
Satyshur, M. D., Layden, E. A., Gowins, J. R., Buchanan, A., & Gollan, J. K. (2018) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-018-0611-7 · Cited by 82
“Fleiss kappa was calculated and the inter-rater reliability for the graduate students was found to be strong (κ=0.82–0.88) (Fleiss, 1971; Fleiss & Cohen, 1973; Randolph, 2016).”
MethodInfluentialSocial Determinants of Health Training in U.S. Primary Care Residency Programs: A Scoping Review
Gard, L. A., Peterson, J., Miller, C. H., Ghosh, N., Youmans, Q. R., Didwania, A. K., Persell, S. D., Jean‐Jacques, M., Ravenna, P. A., O’Brien, M. J., et al. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002491 · Cited by 78
Technology identification from patent texts: A novel named entity recognition method
Puccetti, G., Giordano, V., Spada, I., Chiarello, F., & Fantoni, G. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122160 · Cited by 91
Development of a new welfare assessment protocol for practical application in long‐term dog shelters
Barnard, S., Pedernera, C., Candeloro, L., Ferri, N., Velarde, A., & Villa, P. D. (2016) · Veterinary Record · doi:10.1136/vr.103336 · Cited by 91
“Interobserver reliability was evaluated by means of a Fleiss’ κ index of agreement between observers (Fleiss 1971).”
MethodUnderstanding Positivity Within Dynamic Team Interactions: A Statistical Discourse Analysis
Lehmann‐Willenbrock, N., Chiu, M. M., Lei, Z., & Kauffeld, S. (2016) · Group & Organization Management · doi:10.1177/1059601116628720 · Cited by 88
“We followed a procedure proposed by Fleiss (1971), which allows the measurement of agreement among several raters.”
Distinguishing Subgroups of Narcissists with Latent Class Analysis
Wetzel, E., Leckelt, M., Gerlach, T. M., & Back, M. D. (2016) · European Journal of Personality · doi:10.1002/per.2062 · Cited by 70
“To check the degree of agreement in assigning respondents to narcissism subgroups across models with different predictors, we calculated Fleiss’ Kappa (Fleiss, 1971), an extension of Cohen’s Kappa to multiple raters (in this case models).”
MethodOno, S., Kato, M., Tsuda, M., Miyamoto, S., Abiko, S., Shimizu, Y., & Sakamoto, N. (2018) · Digestion · doi:10.1159/000489454 · Cited by 76
Johnsen, B., Pugdahl, K., Fuglsang‐Frederiksen, A., Kollewe, K., Paracka, L., Dengler, R., Camdessanché, J., Nix, W. A., Liguori, R., Schofield, I. S., et al. (2018) · Clinical Neurophysiology · doi:10.1016/j.clinph.2018.11.021 · Cited by 111
Naismith, G., Smith, L., Barry, S., Munro, J. I., Laird, S., Rankin, K., Morris, A., Winter, J., & Gaya, D. R. (2013) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.12221 · Cited by 76
Buckley, C. J., Sherwin, P. F., Smith, A. P. L., Wolber, J., Weick, S. M., & Brooks, D. J. (2017) · Nuclear Medicine Communications · doi:10.1097/mnm.0000000000000633 · Cited by 69
A Sequential Matching Framework for Multi-Turn Response Selection in Retrieval-Based Chatbots
Wu, Y., Wu, W., Chen, X., Xu, C., Li, Z., & Zhou, M. (2018) · Computational Linguistics · doi:10.1162/coli_a_00345 · Cited by 73
“The Fleiss’ kappa (Fleiss 1971) of the labeling is 0.41, which means that the labelers reached a relatively high agreement in their work.”
BackgroundInfluentialDescription and Reliability of the AOSpine Sacral Classification System
Vaccaro, A. R., Schroeder, G. D., Divi, S. N., Kepler, C. K., Kleweno, C. P., Krieg, J. C., Wilson, J. R., Holstein, J. H., Kurd, M. F., Firoozabadi, R., et al. (2020) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.19.01153 · Cited by 70
ABC of Face Validity for Questionnaire
Patel, N., & Desai, S. (2020) · International Journal of Pharmaceutical Sciences Review and Research · doi:10.47583/ijpsrr.2020.v65i01.025 · Cited by 85
Gaikwad, M., Ahirrao, S. N., Phansalkar, S., & Kotecha, K. V. (2021) · IEEE Access · doi:10.1109/access.2021.3068313 · Cited by 100
“Fleiss’ Kappa measures the agreement for more experts or annotators, while categorizing the data [157].”
BackgroundMMConv: An Environment for Multimodal Conversational Search across Multiple Domains
Liao, L., Long, L. H., Zhang, Z., Huang, M., & Chua, T. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462970 · Cited by 98
“We calculate the averaged weighted kappa value [11] for all dialogue acts over 300 random sampled turns.”
MethodContext-aware Retrieval-based Deep Commit Message Generation
Wang, H., Xia, X., Lo, D., He, Q., Wang, X., & Grundy, J. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3464689 · Cited by 70
“We further use the Fleiss Kappa [17] to measure In summary, our human evaluation shows that CoRec outperforms the state-of-the-art approach.”
MethodDigital literacy scale: Validity and reliability study with the rasch model
Avinç, E., & Doğan, F. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12662-7 · Cited by 159
“In the assessment of kappa, Fleiss’ (1971) recommended criteria were employed.”
Mountz, J. M., Laymon, C. M., Cohen, A. D., Zhang, Z., Price, J. C., Boudhar, S., McDade, E., Aizenstein, H. J., Klunk, W. E., & Mathis, C. A. (2015) · NeuroImage Clinical · doi:10.1016/j.nicl.2015.10.007 · Cited by 77
“Agreement between all 5 raters was determined by Fleiss' kappa (Fleiss, 1971) and was also expressed as overall agreement as well as agreement across all cases (Cicchetti and Feinstein, 1990; Spitzer and Fleiss, 1974).”
Communication and uncertainty reduction during job transfers: Leaving and joining processes
Kramer, M. W. (1993) · Communication Monographs · doi:10.1080/03637759309376307 · Cited by 93
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.