Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 501–600 of 1,118
Reliability and Validity of Utilization Review Criteria
Strumwasser, I., Paranjpe, N. V., Ronis, D. L., Share, D. A., & Sell, L. J. (1990) · Medical Care · doi:10.1097/00005650-199002000-00001 · Cited by 126
Yeung, S., Rinaldo, F., Jopling, J., Liu, B., Mehra, R., Downing, N. L., Guo, M., Bianconi, G. M., Alahi, A., Lee, J., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0087-z · Cited by 125
An analysis of human factors and label accuracy in crowdsourcing relevance judgments
Kazai, G., Kamps, J., & Milić-Frayling, N. (2012) · Information Retrieval · doi:10.1007/s10791-012-9205-0 · Cited by 155
“We can also calculate the level of agreement among the workers using Fleiss' kappa (Fleiss 1971), which ranges from 0 to 1.”
MethodMining software repair models for reasoning on the search space of automated program fixing
Martinez, M., & Monperrus, M. (2015) · Empirical Software Engineering · doi:10.1007/s10664-013-9282-8 · Cited by 125
“The coefficient κ (Kappa) (Cohen et al. 1960; Joseph 1971) measures the confidence in the agreement level by removing the chance factor.”
BackgroundReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation
Zhang, H., Lan, Y., Pang, L., Guo, J., & Cheng, X. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1362 · Cited by 122
“Kappa (Fleiss, 1971) value is presented to demonstrate the consistency of different annotators.”
BackgroundNili, A., Tate, M., Barros, A., & Johnstone, D. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102154 · Cited by 174
“…to code j + n i is the number of items coder 2 assigned to code i + n j is the number of items coder 2 assigned to code j Equation B.6: Fleiss ’ Kappa (Fleiss, 1971) 2014) Gwet (2014) extensively discusses two ways of calculating ICR, including: Gwet ’ s AC 1 and Gwet ’ s AC 2 (weighted…”
MethodInfluentialChurruca, K., Ellis, L. A., Pomare, C., Hogden, A., Bierbaum, M., Long, J. C., Olekalns, A., & Braithwaite, J. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-043982 · Cited by 125
Thornhill, M. H., Sankar, V., Xu, X., Barrett, A. W., High, A. S., Odell, E. W., Speight, P. M., & Farthing, P. M. (2006) · Journal of Oral Pathology and Medicine · doi:10.1111/j.1600-0714.2006.00406.x · Cited by 125
Interobserver reproducibility of histopathological features in stage II breast cancer
Gilchrist, K. W., Kalish, L. A., Gould, V. E., Hirschl, S., Imbriglia, J. E., Levy, W. M., Patchefsky, A. S., Penner, D. W., Pickren, J. W., Roth, J. A., et al. (1985) · Breast Cancer Research and Treatment · doi:10.1007/bf01807642 · Cited by 125
“Using a straightforward extension of the methods of Fleiss (5), the rates of complete agreement among different numbers of pathologists were estimated (Table 1).”
InfluentialShinar, D., Gross, C. R., Mohr, J. P., Caplan, L. R., Price, T. R., Wolf, P. A., Hier, D. B., Kase, C. S., Fishman, I. G., Wolf, C. L., et al. (1985) · Archives of Neurology · doi:10.1001/archneur.1985.04060060059010 · Cited by 125
Acupuncture for Low Back Pain: An Overview of Systematic Reviews
Liu, L., Skinner, M. A., McDonough, S., Mabire, L., & Baxter, D. G. (2015) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2015/328196 · Cited by 154
“Although a considerable variety of conservative therapy alternatives are available for the treatment of LBP, no single modality appears to be superior [2].”
BackgroundCapturing Design Space From a User Perspective: The Repertory Grid Technique Revisited
Hassenzahl, M., & Wessler, R. (2000) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2000.9669070 · Cited by 164
“Interrater agreement (Fleiss, 1971) of the initial categorization performed independently by the three raters was satisfactory ( κ = 0.64, µ = 11.14; p < .01, two-tailed).”
BackgroundInfluentialChan, P., Boriani, S., Fourney, D. R., V, Biagini, R., Dekutoski, M. B., Fehlings, M. G., Ryken, T. C., Gokaslan, Z. L., Vrionis, F. D., Harrop, J. S., et al. (2009) · Spine · doi:10.1097/brs.0b013e3181971283 · Cited by 126
“Fleiss statistic was applied to all interobserver assessments except Enneking metastasis, which is a binary categorical variable.(20) Therefore, Cohen statistic was calculated for the assessment of the interobserver reliability of Enneking metastasis.”
MethodMistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices
Feng, G. C. (2014) · Methodology · doi:10.1027/1614-2241/a000086 · Cited by 144
Awkward encounters of an "other" kind: collective self-presentation and face threat on facebook
Litt, E., Spottswood, E., Birnholtz, J., Hancock, J. T., Smith, M. E., & Reynolds, L. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531646 · Cited by 123
“Inter-rater reliability for types of face threats was acceptable (Cohen's kappa=.73) [16].”
External Validation of the Acoustic Voice Quality Index Version 03.01 With Extended Representativity
Barsties, B., & Maryn, Y. (2016) · Annals of Otology, Rhinology & Laryngology · doi:10.1177/0003489416636131 · Cited by 123
Abd‐El‐Khalick, F., Myers, J. Y., Summers, R., Brunner, J. L., Waight, N., Wahbeh, N., Zeineddin, A. A., & Belarmino, J. J. (2016) · Journal of Research in Science Teaching · doi:10.1002/tea.21339 · Cited by 148
Berrone, P., Ricart, J. E., Duch, A., Bernardo, V., Salvador, J., Piedra-Peña, J., & Planas, M. R. (2019) · Sustainability · doi:10.3390/su11082339 · Cited by 178
“In Table 4, we present Cohen’s kappa [92,93] and its generalization by Conger [94], along with the Brennan and Prediger coefficient [95], Scott/Fleiss’s kappa/pi [96], Gwet’s AC [89,97], and Krippendorff’s alpha coefficient [98] for the case study of the La Chira WWTP.”
MethodInterobserver variation in the assessment of fetal heart rate recordings
Donker, D. K., van Geijn, H. P., & Hasman, A. (1993) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/0028-2243(93)90220-7 · Cited by 139
Corpora for the Conceptualisation and Zoning of Scientific Papers
Liakata, M., Teufel, S., Siddharthan, A., & Batchelor, C. R. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2di6e2euu3as · Cited by 122
“71 (N=3745,n=15,k=3), here reported in terms of the (Fleiss, 1971) κ.”
Named entity recognition in Wikipedia
Balasuriya, D., Ringland, N., Nothman, J., Murphy, T., & Curran, J. R. (2009) · doi:10.3115/1699765.1699767 · Cited by 120
“By this standard, our three annotators were in strong agreement prior to discussion, with our Fleiss’ κ values all greater than 0.81.”
MethodInfluentialDetecting Hallucinated Content in Conditional Neural Sequence Generation
Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., & Ghazvininejad, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.120 · Cited by 122
Patterns of intraocular pressure elevation after aqueous humor outflow obstruction in rats.
Jia, L., Cepurna, W. O., Johnson, E. C., & Morrison, J. C. (2000) · PubMed · Cited by 122
Aickin, M. (1990) · Biometrics · doi:10.2307/2531434 · Cited by 122
Riddle, D. L., Freburger, J. K., & North American Orthopaedic Rehabilitation Research Network (2002) · Physical Therapy · doi:10.1093/ptj/82.8.772 · Cited by 121
MRI in the diagnosis of sporadic Creutzfeldt–Jakob disease: a study on inter-observer agreement
Tschampa, H. J., Kallenberg, K., Urbach, H., Meißner, B., Nicolay, C., Kretzschmar, H. A., Knauth, M., & Zerr, I. (2005) · Brain · doi:10.1093/brain/awh575 · Cited by 122
“Inter-observer agreement in MRI analysis To determine the degree of agreement between the observers, the kappa coefficient (k) was calculated for each pair of observers as well as the overall k (all observers combined) (Fleiss, 1971).”
MethodError weighted semi-coupled hidden Markov model for audio-visual emotion recognition
Lin, J.-C., Wu, C.-H., & Wei, W.-L. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2171334 · Cited by 121
“Fleiss’ kappa can work for any number of raters giving categorical ratings to a fixed number of items [46].”
BackgroundHow transformational leadership works during team interactions: A behavioral process analysis
Lehmann‐Willenbrock, N., Meinecke, A. L., Rowold, J., & Kauffeld, S. (2015) · The Leadership Quarterly · doi:10.1016/j.leaqua.2015.07.003 · Cited by 193
“To calculate inter-rater reliability for theinteractioncoding, wefollowedaprocedure proposed by Fleiss (1971) by which κ can be generalized to several coders and obtained a value of Fleiss' κ = .81.”
A New Subtype Classification of Ischemic Stroke Based on Treatment and Etiologic Mechanism
Han, S. W., Kim, S. H., Lee, J. Y., Chu, C. K., Yang, J. H., Shin, H. Y., Nam, H. S., Lee, B. I., & Heo, J. H. (2006) · European Neurology · doi:10.1159/000098059 · Cited by 121
“Three neurologists independently reviewed all available clinical, imaging and laboratory data of each patient, and then they classified the patients according to the new and the TOAST classification systems. The raters have had experience with the TOAST classification for more than 3 years and had not been involved in…”
MethodHashimoto, C., Torisawa, K., Kloetzer, J., Sano, M., Varga, I., Oh, J., & Kidawara, Y. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1093 · Cited by 121
“The Kappa (Fleiss, 1971) of their judgments was 0.67 (substantial agreement (Lan-dis and Koch, 1977)).”
BackgroundInfluentialHow Robust Are Multirater Interrater Reliability Indices to Changes in Frequency Distribution?
Quarfoot, D., & Levine, R. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1141708 · Cited by 134
“These indices were Fleiss’s kappa, Conger’s kappa, Krippendorff ’s alpha, the Brennan-Prediger coefficient, and Gwet’s AC2 index (Fleiss 1971; Conger 1980; Brennan and Prediger 1981; Krippendorff 2012; Gwet 2014).”
MethodLanglois, Y., Roederer, G. O., Chan, A., Phillips, D. J., Beach, K. W., Martin, D., Chikos, P. M., & Strandness, D. E. (1983) · Ultrasound in Medicine & Biology · doi:10.1016/0301-5629(83)90109-6 · Cited by 120
Visual Literacy in Bloom: Using Bloom’s Taxonomy to Support Visual Learning Skills
Arneson, J. B., & Offerdahl, E. G. (2018) · CBE—Life Sciences Education · doi:10.1187/cbe.17-08-0178 · Cited by 120
“A Fleiss’s kappa, which can be used to compare the observed agreement of two or more coders with the expected agreement while accounting for the likelihood a category was coded by chance (Fleiss, 1971), was also calculated (κ = 0.”
MethodInfluentialSynthesis of diagnostic quality cancer pathology images by generative adversarial networks
Levine, A. B., Peng, J., Farnell, D., Nursey, M., Wang, Y., Naso, J. R., Ren, H., Farahani, H., Chen, C., Chiu, D., et al. (2020) · The Journal of Pathology · doi:10.1002/path.5509 · Cited by 120
“Analysis of inter-rater agreement demonstrated strong concordance (although lower than the concordance between a rater’s diagnosis and the true diagnosis) for histotype classification, with a Fleiss’ kappa [37] of 0.”
BackgroundMethodAssessment of agreement among several raters formulating multiple diagnoses
Mezzich, J. E., KRAEMER, H. C., Worthington, D. R. L., & Coffman, G. A. (1981) · Journal of Psychiatric Research · doi:10.1016/0022-3956(81)90011-x · Cited by 122
Goulet, J. L., Fultz, S. L., McGinnis, K. A., & Justice, A. C. (2005) · AIDS · doi:10.1097/01.aids.0000192077.11067.e5 · Cited by 119
“Kappa statistics, a measure of chancecorrected agreement [24], revealed that electronic and chart review as well as laboratory data were in moderate agreement [25].”
BackgroundIslam, S., & Zibran, M. F. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.08.030 · Cited by 143
“We also measure the degree of inter-rater agreement in terms of Fleiss-κ (Fleiss, 1971) value.”
MethodInfluentialAtypical depression: clinical aspects and noradrenergic function
(1995) · American Journal of Psychiatry · doi:10.1176/ajp.152.1.31 · Cited by 119
The user action framework: a reliable foundation for usability engineering support tools
Andre, T. S., Hartson, H. R., Belz, S. M., & McCreary, F. A. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2000.0441 · Cited by 147
Psychiatric diagnosis across cultural boundaries
(1985) · American Journal of Psychiatry · doi:10.1176/ajp.142.7.798 · Cited by 119
Trends in Newspaper Coverage of Mental Illness in Canada: 2005–2010
Whitley, R., & Berry, S. (2013) · The Canadian Journal of Psychiatry · doi:10.1177/070674371305800208 · Cited by 118
Bader, M. D. M., Mooney, S. J., Lee, Y., Sheehan, D., Neckerman, K. M., Rundle, A., & Teitler, J. O. (2014) · Health & Place · doi:10.1016/j.healthplace.2014.10.012 · Cited by 125
“For items with five or fewer response categories or categorical response categories, we assessed inter-rater reliability (IRR) using Fliess’ κ statistic (Fleiss, 1971).”
Pieper, D., Puljak, L., González‐Lorenzo, M., & Minozzi, S. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.12.004 · Cited by 130
“We calculated Fleiss’ Kappa (k) as a measure of interrater reliability (IRR) for each AMSTAR 2 question and for each ROBIS domain and signaling question between all four reviewers [19].”
MethodWang, M., Shen, L. Q., Pasquale, L. R., Petrakos, P., Formica, S., Boland, M. V., Wellik, S. R., Moraes, C. G. D., Myers, J. S., Saeedi, O. J., et al. (2019) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.18-25568 · Cited by 129
Tomlinson, E. C., Schnackenberg, A. K., Dawley, D., & Ash, S. R. (2020) · Journal of Organizational Behavior · doi:10.1002/job.2448 · Cited by 118
Late Radiation Injury to the Temporal Lobes: Morphologic Evaluation at MR Imaging
Chan, Y.-L., Leung, S. F., King, A. D., Choi, P. H. K., & Metreweli, C. (1999) · Radiology · doi:10.1148/radiology.213.3.r99dc07800 · Cited by 118
O'Haire, C., & Gibbons, P. F. (2000) · Manual Therapy · doi:10.1054/math.1999.0203 · Cited by 118
“In this study, Kg was tested for signi®cant di erence from chance agreement using the standard error (SEo) statistic (Fleiss 1971; Haas 1991a).”
BackgroundMethodBeckstead, J. W. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.04.014 · Cited by 131
“went on to interpret/ evaluate k* as though it were kappa using standards recommended by Fleiss (1981) and also provided a table of standards for relating k* to the item-level CVI.”
BackgroundMethodInfluentialEarly Cerebral Lesions in Cytomegalovirus Infection: Prenatal MR Imaging
Doneda, C., Parazzini, C., Righini, A., Rustico, M., Tassis, B., Fabbri, E., Arrigoni, F., Consonni, D., & Triulzi, F. M. (2010) · Radiology · doi:10.1148/radiol.10090749 · Cited by 127
Reports of paternal alcoholism: Reliability across siblings
Sher, K. J., & Descutner, C. J. (1986) · Addictive Behaviors · doi:10.1016/0306-4603(86)90005-5 · Cited by 118
Rogart, J. N., Jain, D., Siddiqui, U. D., Oren, T., Lim, J. K., Jamidar, P. A., & Aslanian, H. R. (2008) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2008.04.035 · Cited by 117
Intercoder reliability indices: disuse, misuse, and abuse
Feng, G. C. (2013) · Quality & Quantity · doi:10.1007/s11135-013-9956-8 · Cited by 142
Colli, A., & Lingiardi, V. (2009) · Psychotherapy Research · doi:10.1080/10503300903121098 · Cited by 118
“Interrater reliability in this case was evaluated in 2,984 utterances (2,984 for patient and 2,984 for therapist) using Cohen's kappa (Fleiss, 1981 ).”
MethodGueler, M. S., & Schneider, S. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.04.061 · Cited by 176
Online peer feedback patterns of success and failure in argumentative essay writing
Kerman, N. T., Noroozi, O., Banihashem, S. K., Karami, M., & Biemans, H. J. A. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2022.2093914 · Cited by 116
“Since there were more than two coders, Fleiss ’ Kappa statistic was used to determine the inter-rater reliability between coders (Fleiss, 1971), and the agreement was found to be 75% ( Fleiss ’ Kappa = 0.75 [IC 95%: 0.70-0.81]; z = 26.08; p < 0.001 ) indicating signi fi cant agreement between the…”
MethodInfluentialAnders, T. F., Keener, M. A., & Kraemer, H. (1985) · Sleep · doi:10.1093/sleep/8.3.193 · Cited by 117
Patients' attitudes toward hallucinations
(1993) · American Journal of Psychiatry · doi:10.1176/ajp.150.4.584 · Cited by 117
DSM-III personality diagnosis in bulimia
Levin, A. P., & Hyler, S. E. (1986) · Comprehensive Psychiatry · doi:10.1016/0010-440x(86)90068-4 · Cited by 117
Behutiye, W. N., Rodríguez, P., Oivo, M., & Tosun, A. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2016.10.004 · Cited by 116
Chakrabarty, S., & Bass, A. E. (2013) · Journal of Business Ethics · doi:10.1007/s10551-013-1963-0 · Cited by 145
“73 indicating substantial agreement) (Fleiss 1971; Landis and Koch 1977; Shrout and Fleiss 1979).”
InfluentialBurden of emergency conditions and emergency care usage: new estimates from 40 countries
Chang, C. Y., Abujaber, S., Reynolds, T. A., Camargo, C. A., & Obermeyer, Z. (2016) · Emergency Medicine Journal · doi:10.1136/emermed-2016-205709 · Cited by 126
Leveraging Automated Sentiment Analysis in Software Engineering
Islam, M. R., & Zibran, M. F. (2017) · 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.9 · Cited by 111
Summarising good practice guidelines for data extraction for systematic reviews and meta-analysis
Taylor, K. S., Mahtani, K. R., & Aronson, J. K. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111651 · Cited by 115
“Then before comparing the extracted data and seeking a consensus, the extent to which coded (categorical) data extracted by two different reviewers are consistent may be measured using kappa statistics,(1,2,12,15) or Fleis kappa statistics, when more than two people have extracted the data.(16) Formal comparisons are…”
BackgroundMethodBenchmarking Kappa: Interrater Agreement in Software Process Assessments
Emam, K. E. (1999) · Empirical Software Engineering · doi:10.1023/a:1009820201126 · Cited by 120
“For instance, Kappa has been used to evaluate the agreement in identifying mental disorders, such as depression, neurosis, and schizophrenia (Fleiss, 1971).”
BackgroundMethodIllusions of Meaning in the Ipsative Assessment of Children's Ability
McDermott, P. A., Fantuzzo, J. W., Glutting, J. J., Watkins, M. W., & Baggaley, A. R. (1992) · The Journal of Special Education · doi:10.1177/002246699202500407 · Cited by 116
“Classificatory stability for observed strengths and weaknesses was assessed using Fleiss's (1971) extension of coefficient kappa to partial kappa via computer program CONGRU (Watkins & McDermott, 1979).”
Evaluative criteria for psychiatric classification.
Blashfield, R. K., & Dragūns, J. G. (1976) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.85.2.140 · Cited by 116
Chance and interrater agreement on manuscripts.
Watkins, M. W. (1979) · American Psychologist · doi:10.1037/0003-066x.34.9.796 · Cited by 116
Assessing the inter-rater agreement for ordinal data through weighted indexes
Marasini, D., Quatto, P., & Ripamonti, E. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280214529560 · Cited by 115
“Fleiss (1971) put forth a generalization of kappa in the case of multiple observers, but both Cohen’s and Fleiss’ kappa could have a paradoxical behavior, which may lead to a difficult interpretation of their magnitude.”
BackgroundKlotz, M., Kemper, T., Geiß, C., Esch, T., & Taubenböck, H. (2016) · Remote Sensing of Environment · doi:10.1016/j.rse.2016.03.001 · Cited by 127
“…that rely on the entire error matrix and consider both types of errors: Kappa (Galton, 1892) presents a discrete multivariate statistic to test if binary (Cohen, 1960) or multi-categorical (Fleiss, 1971) classifications are significantly different from one another or respective reference data.”
Histological and reference system for the analysis of mouse intervertebral disc
Tam, V., Chan, W. C. W., Leung, V., Cheah, K. S. E., Cheung, K., Sakai, D., McCann, M. R., Bedore, J., Séguin, C. A., & Chan, D. T.-M. (2017) · Journal of Orthopaedic Research® · doi:10.1002/jor.23637 · Cited by 115
“Inter-observer agreement was calculated by using Fleiss’ Kappa.39 The students’ T-test was used to determine significance between the Ent1 KO, Ccn2 cKO, and WT mice.”
MethodTowards Persona-Based Empathetic Conversational Models
Zhong, P., Zhang, C., Wang, H., Liu, Y., & Miao, C. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.531 · Cited by 109
“The inter-annotator agreement, measured by Fleiss’ kappa (Fleiss, 1971), for sentiment and empathy are 0.”
Humphreys, B. L., McCray, A. T., & Cheh, M. L. (1997) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1997.0040484 · Cited by 115
Hurdles to Progress in Long-form Question Answering
Krishna, K., Roy, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.393 · Cited by 114
Brackenridge, C. H., Bishopp, D., Moussalli, S., & Tapp, J. (2008) · International Journal of Sport and Exercise Psychology · doi:10.1080/1612197x.2008.9671881 · Cited by 139
“Cohen’s (1960) Kappa was 0.85, which is considered acceptable (Fleiss, 1971) and indicates an agreement of 92%.”
BackgroundInfluentialTonkin, M. A., Tolerton, S. K., Quick, T., Harvey, I., Lawson, R. D., Smith, N. C., & Oberg, K. C. (2013) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2013.03.019 · Cited by 114
A Comparison of Reliability Coefficients for Ordinal Rating Scales
de Raadt, A., Warrens, M. J., Bosker, R. J., & Kiers, H. A. L. (2021) · Journal of Classification · doi:10.1007/s00357-021-09386-5 · Cited by 112
“An extension of Scott’s pi to multiple raters is presented in Fleiss (1971).”
BackgroundHelping Developers Help Themselves: Automatic Decomposition of Code Review Changesets
Barnett, M., Bird, C., Brunet, J., & Lahiri, S. K. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.35 · Cited by 185
“Since we used more than two raters for the transcriptions we used Fleiss’ Kappa [16] on the individually coded responses to determine inter-rater reliability.”
MethodKnowledge and the Sequential Processes of Negotiation: A Markov Chain Analysis of Response-in-Kind
Weingart, L. R., Prietula, M. J., Hyder, E. B., & Genovese, C. R. (1999) · Journal of Experimental Social Psychology · doi:10.1006/jesp.1999.1378 · Cited by 123
“Prior to reconciliation, intercoder reliability (Cohen’s kappa, 1960) was computed for each category from a subset of dyads (28 dyads, 4302 codes) and ranged from .75 to .89 (p , .001 for all categories; Fleiss, 1971).”
MethodMeasurement of reliability for categorical data in medical research
Kraemer, H. C. (1992) · Statistical Methods in Medical Research · doi:10.1177/096228029200100204 · Cited by 114
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Cabitza, F., Campagner, A., & Basile, V. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25840 · Cited by 114
“As we said above, a perspectivist approach can be advocated especially for the annotation of those phenomena that we would consider as “highly subjective”, from a common sense stance, that is affected by high observer variability, as measured with some inter-rater agreement metrics, like kappa [22], alpha [31] or rho…”
BackgroundMethodPutting the Kappa Statistic to Use
Nichols, T. R., Wisner, P. M., Cripe, G., & Gulabchand, L. (2010) · The Quality Assurance Journal · doi:10.1002/qaj.481 · Cited by 111
Tan, R. H., Kril, J. J., Fatima, M., McGeachie, A., McCann, H., Shepherd, C., Forrest, S. L., Affleck, A., Kwok, J. B. J., Hodges, J. R., et al. (2015) · Brain · doi:10.1093/brain/awv220 · Cited by 112
Intrafocal (Kapandji) pinning of distal radius fractures with and without external fixation
Trumble, T. E., Wagner, W., Hanel, D. P., Vedder, N. B., & Gilbert, M. (1998) · The Journal Of Hand Surgery · doi:10.1016/s0363-5023(05)80455-1 · Cited by 113
Tomassen, P., Newson, R. B., Hoffmans, R., Lötvall, J., Cardell, L. O., Gunnbjörnsdóttir, M., Thilsing, T., Matricardi, P., Krämer, U., Makowska, J. S., et al. (2011) · Allergy · doi:10.1111/j.1398-9995.2010.02503.x · Cited by 113
“Cohen’s kappa is a widely accepted measure to assess chance-corrected agreement (9) but it has been argued that in questionnaire development for assessing symptoms in population-based studies (where the prevalence of the outcome is low) survey items should not be rejected on the basis of kappa alone (8).”
BackgroundHasselmann, M. H., & Reichenheim, M. E. (2003) · Cadernos de Saúde Pública · doi:10.1590/s0102-311x2003000400030 · Cited by 147
Jeong, E. Y., Kim, H. L., Ha, E. J., Park, S. Y., Cho, Y. J., & Han, M. (2018) · European Radiology · doi:10.1007/s00330-018-5772-9 · Cited by 112
Comparative analyses of SPI and SPEI as drought assessment tools in Tigray Region, Northern Ethiopia
Tefera, A. S., Ayoade, J. O., & Bello, N. J. (2019) · SN Applied Sciences · doi:10.1007/s42452-019-1326-2 · Cited by 120
Haahr-Pedersen, I., Ershadi, A. E., Hyland, P., Hansen, M., Perera, C., Sheaf, G., Bramsen, R. H., Spitz, P., & Vallières, F. (2020) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2020.104589 · Cited by 112
Exploring Explainability: A Definition, a Model, and a Knowledge Catalogue
Chazette, L., Brunotte, W., & Speith, T. (2021) · 2021 IEEE 29th International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00025 · Cited by 105
Three-point checklist of dermoscopy: an open internet study
Zalaudek, I., Argenziano, G., Soyer, H. P., Corona, R., Sera, F., Blum, A., Braun, R. P., Cabo, H., Ferrara, G., Kopf, A. W., et al. (2005) · British Journal of Dermatology · doi:10.1111/j.1365-2133.2005.06983.x · Cited by 115
Inter-examiner and intra-examiner reliability of the standing flexion test
Vincent-Smith, B., & Gibbons, P. F. (1999) · Manual Therapy · doi:10.1054/math.1999.0173 · Cited by 112
Markov Chain Monte Carlo Methods for Fitting Spatiotemporal Stochastic Models in Plant Epidemiology
Gibson, G. J. (1997) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00061 · Cited by 151
“Three measures of aggregation were considered: a join count statistic (Cliff and Ord, 1981), the aggregation parameter of a beta-binomial distribution fitted to quadrat count frequencies (Hughes and Madden, 1993) and the intraclass correlation (Fleiss, 1971) calculated from quadrat count frequencies.”
BackgroundMethodInfluentialHattori, M., & Oaksford, M. (2007) · Cognitive Science · doi:10.1080/03640210701530755 · Cited by 125
“(8) This index has been used as a measure of " response agreement " in psychology (Cohen, 1960, 1968; Fleiss, 1971; Light, 1971), " observer agreement " in clinical medicine (Landis & Koch, 1977), and the " accuracy " of weather forecasting (called the Heidke skill score) especially in America…”
MethodAnderson, S. W., Soto, J. A., Lucey, B. C., Ozonoff, A., Jordan, J. D., Ratevosian, J., Ulrich, A. S., Rathlev, N. K., Mitchell, P. M., Rebholz, C. M., et al. (2009) · American Journal of Roentgenology · doi:10.2214/ajr.09.2336 · Cited by 112
“1, R Foundation for Statistical Computing, Vienna University of Technology, Vienna, Austria) [12].”
BackgroundComputed Tomographic Angiography for the Evaluation of Carotid Artery Stenosis
Leclerc, X., Godefroy, O., Pruvo, J., & Leys, D. (1995) · Stroke · doi:10.1161/01.str.26.9.1577 · Cited by 112
Zhang, M., Zhang, L., Zou, J., Yao, C., Xiao, H., Liu, Q., Wang, J., Wang, D., Wang, C., & Guo, Z. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp295 · Cited by 116
Country-Specific Associations Made by Consumers: A Dual-Coding Theory Perspective
Herz, M. F., & Diamantopoulos, A. (2013) · Journal of International Marketing · doi:10.1509/jim.13.0004 · Cited by 130
The Indian Spontaneous Expression Database for Emotion Recognition
Happy, S. L., Patnaik, P., Routray, A., & Guha, R. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2498174 · Cited by 123
“The reliability of agreement between four raters is also evaluated by using Fleiss’s kappa [42].”
MethodThree risk of bias tools lead to opposite conclusions in observational research synthesis
Losilla, J.-M., Oliveras, I., Marin-Garcia, J. A., & Vives, J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.021 · Cited by 111
Creative thinking in mathematics curriculum: An analytic framework
Hadar, L. L., & Tirosh, M. (2019) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2019.100585 · Cited by 158
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.