Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 401–500 of 1,118
A ranking approach to keyphrase extraction
Jiang, X., Hu, Y., & Li, H. (2009) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1571941.1572113 · Cited by 158
“We used Fleiss’ kappa, a commonly used measure, to asses the degree of agreement between the annotators [7]:”
MethodDevelopment and Evaluation of the Dutch Clinician-Administered PTSD Scale for DSM-5 (CAPS-5)
Boeschoten, M. A., van der Aa, N., Bakker, A., ter Heide, F. J. J., Hoofwijk, M. C., Jongedijk, R. A., van Minnen, A., Elzinga, B. M., & Olff, M. (2018) · European Journal of Psychotraumatology · doi:10.1080/20008198.2018.1546085 · Cited by 168
“Interrater reliability of the categorical PTSD diagnosis was evaluated by calculating Fleiss’s kappa (Fleiss, 1971), which was used because the first and”
MethodProactive Human-Machine Conversation with Explicit Conversation Goal
Wu, W., Guo, Z., Zhou, X., Wu, H. R., Zhang, X., Lian, R., & Wang, H. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1369 · Cited by 244
“All human evaluations are conducted by three persons, where the agreement ratio (Fleiss’ kappa (Fleiss et al., 1971)) is from 0.37 to 0.86, with the lowest agreement on multi-turn coherence and others all above 0.6.”
InfluentialGonzález-Valero, G., Ortega, F. Z., Ubago-Jiménez, J. L., & Puertas-Molero, P. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16224394 · Cited by 219
“To establish methodological quality of the study, reliability of selection and detection for more than two of the evaluators was determined using the Kappa de Fleiss statistical measure (Kf) [29], whilst the Cohen’s Kappa statistical index (Kc) was used to examine the coding of two authors [30].”
MethodA practical outcome scale for paediatric head injury
Crouchman, M., Rossiter, L., Colaco, T., & Forsyth, R. (2001) · Archives of Disease in Childhood · doi:10.1136/adc.84.2.120 · Cited by 157
Leys, D., Soetaert, G., Petit, H., Fauquette, A., Pruvo, J.-P., & Steinling, M. (1990) · Archives of Neurology · doi:10.1001/archneur.1990.00530050040010 · Cited by 147
Deep learning similarities from different representations of source code
Tufano, M., Watson, C., Bavota, G., Di Penta, M., White, M., & Poshyvanyk, D. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196431 · Cited by 158
“In order to statistically assess the evaluators’ agreement, we compute the Fleiss’ kappa [24].”
MethodFine-Grained Sentiment Analysis with Structural Features
Zirn, C., Niepert, M., Stuckenschmidt, H., & Strube, M. (2011) · MADOC (University of Mannheim) · Cited by 176
Koehler, J. W., Miller, A. D., Miller, C. R., Porter, B. F., Aldape, K., Beck, J. A., Brat, D. J., Cornax, I., Corps, K. N., Frank, C., et al. (2018) · Journal of Neuropathology & Experimental Neurology · doi:10.1093/jnen/nly085 · Cited by 165
Holm, J. B., Humphrys, M. S., Robinson, C. K., Settles, M. L., Ott, S. H., Fu, L., Yang, H., Gajer, P., He, X., McComb, E. J., et al. (2019) · mSystems · doi:10.1128/msystems.00029-19 · Cited by 174
“Agreement of within-subject assigned CSTs between methods was determined using Fleiss’ kappa statistic (22) (R package irr, v 0.”
A systematic review of placement-related attrition in nurse education
Eick, S., Williamson, G. R., & Heath, V. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.12.004 · Cited by 145
What Level of Quality Can Neural Machine Translation Attain on Literary Text?
Toral, A., & Way, A. (2018) · Machine translation · doi:10.1007/978-3-319-91241-7_12 · Cited by 194
Valla, J., Bergeron, L., Bérubé, H., Gaudet, N., & St‐Georges, M. (1994) · Research on Child and Adolescent Psychopathology · doi:10.1007/bf02168082 · Cited by 175
Hoonhorst, M. H. J., Nijland, R. H. M., van den Berg, J. S. P., Emmelot, C. H., Kollen, B. J., & Kwakkel, G. (2015) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2015.06.009 · Cited by 170
Byrne, M. L., O’Brien-Simpson, N. M., Reynolds, E. C., Walsh, K. A., Laughton, K., Waloszek, J. M., Woods, M. J., Trinder, J., & Allen, N. B. (2013) · Brain, Behavior, and Immunity · doi:10.1016/j.bbi.2013.08.010 · Cited by 144
“Approximately 20% of interviews were double-scored by another researcher, and inter-rater reliability was calculated by the author at the item level, including symptoms and diagnoses, using the kappa (κ) statistic (Cohen, 1960; Fleiss, 1971).”
Amato, M. P., Fratiglioni, L., Groppi, C., Siracusa, G., & Amaducci, L. (1988) · Archives of Neurology · doi:10.1001/archneur.1988.00520310052017 · Cited by 144
Modelling patterns of agreement and disagreement
Agresti, A. (1992) · Statistical Methods in Medical Research · doi:10.1177/096228029200100205 · Cited by 144
EliXR: an approach to eligibility criteria extraction and representation
Weng, C., Wu, X., Luo, Z., Boland, M. R., Theodoratos, D., & Johnson, S. B. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000321 · Cited by 143
“The interrater agreement was calculated using Fleiss ’ k , 24 which can measure the interrater agreement among more than two raters.”
MethodMacapagal, K. R., Coventry, R., Arbeit, M. R., Fisher, C. B., & Mustanski, B. S. (2016) · Archives of Sexual Behavior · doi:10.1007/s10508-016-0784-5 · Cited by 155
“87 for theguardianpermissioncodes, indicated excellent intercoder reliability (Dedoose, 2015; Fleiss, 1971).”
AraSenTi-Tweet: A Corpus for Arabic Sentiment Analysis of Saudi Tweets
Al-Twairesh, N., Al-Khalifa, H. S., Al‐Salman, A. S., & Al-Ohali, Y. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.10.094 · Cited by 149
“We calculated Fleiss’s Kappa for the 14,806 tweets that were annotated by three annotators with the following classes: positive, negative, mixed, neutral, and indeterminate.”
MethodSentiCR: A customized sentiment analysis tool for code review interactions
Ahmed, T., Bosu, A., Iqbal, A., & Rahimi, S. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115623 · Cited by 194
“We used Fleiss’ Kappa [11] (useful for more than two raters) to measure the level of agreement among the three raters. κ = 0 .”
MethodComparing pre-trained language models for Spanish hate speech detection
Plaza-del-Arco, F. M., Molina-González, M. D., López, L. A. U., & Valdivia, M. T. M. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114120 · Cited by 189
Zagalaz, J. C., Zafra, M. S., Sanabrias-Moreno, D., González-Valero, G., Lara-Sánchez, A. J., & Sánchez, M. L. Z. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.569348 · Cited by 168
“In order to establish the methodological quality of the present study, reliability was determined according to the detection and selection of the Fleiss’ Kappa (Fk) statistical index for more than two evaluators (Fleiss, 1971).”
MethodMattonen, S. A., Palma, D. A., Johnson, C., Louie, A. V., Landis, M. S., Rodrigues, G., Chan, I. Y. M., Etemad‐Rezai, R., Yeung, T. P. C., Senan, S., et al. (2015) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2015.12.369 · Cited by 151
“Fleiss' kappa was used to measure the reliability of agreement in all observations, across all six raters (20, 21).”
MethodFeuding Families and Former Friends: Unsupervised Learning for Dynamic Fictional Relationships
Iyyer, M., Guha, A., Chaturvedi, S., Boyd-Graber, J. L., & Daumé, H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1180 · Cited by 143
Real Time Quantification of Dangerousity in Football Using Spatiotemporal Tracking Data
Link, D., Lang, S., & Seidenschwarz, P. (2016) · PLOS ONE · doi:10.1371/journal.pone.0168768 · Cited by 138
Extracting brand information from social networks: Integrating image, text, and social tagging data
Klostermann, J., Plumeyer, A., Böger, D., & Decker, R. (2018) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2018.08.002 · Cited by 162
Repeatability and interobserver error of digit ratio (2D:4D) measurements made by experts
Voracek, M., Manning, J. T., & Dressler, S. G. (2006) · American Journal of Human Biology · doi:10.1002/ajhb.20581 · Cited by 143
“These investigators’ agreement on this issue was quantified, using the generalized kappa coefficient of Fleiss (1971).”
MethodAharonovich, E., Liu, X., Nunes, E., & Hasin, D. S. (2002) · American Journal of Psychiatry · doi:10.1176/appi.ajp.159.9.1600 · Cited by 141
Developing a test collection for biomedical word sense disambiguation.
Weeber, M., Mork, J. G., & Aronson, A. R. (2001) · Europe PMC (PubMed Central) · Cited by 140
Gormsen, L. C., Haraldsen, A., Kramer, S., Dias, A. H., Kim, W. Y., & Borghammer, P. (2016) · EJNMMI Research · doi:10.1186/s13550-016-0207-6 · Cited by 140
“Interobserver agreement on the diagnosis of CS and patterns of (18)F-FDG uptake was evaluated by estimating the overall agreement and the kappa statistic for multiple raters (experts) per subject (patient), a method put forward by Fleiss in 1971 [19].”
MethodImproving breast cancer diagnostics with deep learning for MRI
Witowski, J. S., Heacock, L. C., Reig, B., Kang, S. K., Lewin, A. A., Pysarenko, K. M., Patel, S. J., Samreen, N., Rudnicki, W., Łuczyńska, E., et al. (2022) · Science Translational Medicine · doi:10.1126/scitranslmed.abo4802 · Cited by 158
CD8+ and FOXP3+ Tumor-Infiltrating T Cells Before and After Chemoradiotherapy for Rectal Cancer
Shinto, E., Hase, K., Hashiguchi, Y., Sekizawa, A., Ueno, H., Shikina, A., Kajiwara, Y., Kobayashi, H., Ishiguro, M., & Yamamoto, J. (2014) · Annals of Surgical Oncology · doi:10.1245/s10434-014-3584-y · Cited by 158
Marlovits, S., Striessnig, G., Kutscha-Lissberg, F., Resinger, C., Aldrian, S. M., Vécsei, V., & Trattnig, S. (2004) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-004-0535-3 · Cited by 137
“To evaluate the validity of the scoring system, all MRI scans were read by two independent radiologists and the interobserver variability was determined using j-statistics [9].”
MethodQuantifying the development of agreement among experts in Delphi studies
Meijering, J. V., Kampen, J. K., & Tobi, H. (2013) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2013.01.003 · Cited by 172
Choe, J., Hwang, H. J., Seo, J. B., Lee, S. M., Yun, J., Kim, M.-J., Jeong, J., Lee, Y., Jin, K., Park, R., et al. (2021) · Radiology · doi:10.1148/radiol.2021204164 · Cited by 159
Statistical Tools for Measuring Agreement
Lin, L., Hedayat, A., & Wu, W. (2011) · Springer eBooks · doi:10.1007/978-1-4614-0562-7 · Cited by 149
Explaining the Advocacy Agenda: Insights from the Human Security Network
Carpenter, C., Duygulu, Ş., Montgomery, A. H., & Rapp, A. (2014) · International Organization · doi:10.1017/s0020818313000453 · Cited by 138
“Fleiss 1971.”
Testing the Difference of Correlated Agreement Coefficients for Statistical Significance
Gwet, K. L. (2015) · Educational and Psychological Measurement · doi:10.1177/0013164415596420 · Cited by 144
“Let us consider consider the generalized kappa statistic of Fleiss (1971) often used to quantify the extent of agreement among multiple raters.”
MethodHefner, V., & Wilson, B. J. (2013) · Communication Monographs · doi:10.1080/03637751.2013.776697 · Cited by 134
“To assess agreement among coders, we calculated intercoder reliability using Fleiss’ Kappa, which is an extension of Cohen’s Kappa that allows for more than two coders and corrects for agreements based on chance (Fleiss, 1971; Riffe, Lacy, & Fico, 2005).”
MethodBootvong, K., Liu, Z., McGrath, C. P. J., Hägg, U., Wong, R., Bendeus, M., & Yeung, S. (2010) · European Journal of Orthodontics · doi:10.1093/ejo/cjp159 · Cited by 137
“Comparison analysis was determined by calculating the mean directional differences (MDDs) and standardized directional differences (SDDs; Fleiss, 1971).”
MethodBarak, M., & Levenberg, A. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.04.003 · Cited by 137
Mortality on Mount Everest, 1921-2006: descriptive study
Firth, P. G., Zheng, H., Windsor, J. S., Sutherland, A., Imray, C., Moore, G. W. K., Semple, J. L., Roach, R. C., & Salisbury, R. A. (2008) · BMJ · doi:10.1136/bmj.a2654 · Cited by 137
“We used the Fleiss’ kappa method to analyse the inter-rater agreement for classification of death.”
Jiang, Y., Nishikawa, R. M., Schmidt, R. A., Toledano, A. Y., & Doi, K. (2001) · Radiology · doi:10.1148/radiol.220001257 · Cited by 135
Supporting Collaborative Learning and E-Discussions Using Artificial Intelligence Techniques
Mclaren, B. M., Scheuer, O., & Mikšátko, J. (2010) · International Journal of Artificial Intelligence in Education · doi:10.3233/jai-2010-0001 · Cited by 136
“and Fleiss’s κ (1971) applied when more than two coders did the coding.”
MethodNeural Network-based Detection of Self-Admitted Technical Debt: From Performance to Explainability
Ren, X., Xing, Z., Xia, X., Lo, D., Wang, X., & Grundy, J. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3324916 · Cited by 178
“The Fleiss Kappa coefficient of our model and the five participants’ SATD/non-SATD labels is 0.683, which is also a substantial agreement.”
MethodInfluentialAmgad, M., Atteya, L. A., Hussein, H., Mohammed, K. H., Hafiz, E., Elsebaie, M. A. T., Alhusseiny, A. M., AlMoslemany, M. A., Elmatboly, A. M., Pappalardo, P. A., et al. (2022) · GigaScience · doi:10.1093/gigascience/giac037 · Cited by 115
“We used thresholds set by Fleiss for defining slight,fair,moderate,substantial,and near-perfect agreement [61].”
Hiraoka, N., Ino, Y., Sekine, S., Tsuda, H., Shimada, K., Kosuge, T., Závada, J., Yoshida, M., Yamada, K., Koyama, T., et al. (2010) · British Journal of Cancer · doi:10.1038/sj.bjc.6605854 · Cited by 142
“According to the widely used statistical chart that grades the strength of agreement (Fleiss, 1971; Landis and Koch, 1977) into 6 categories (poor (k-value, o0.”
MethodThe Elephant in the Schoolhouse: The Role of Propinquity in School Staff Interactions about Teaching
Spillane, J. P., Shirrell, M., & Sweet, T. M. (2017) · Sociology of Education · doi:10.1177/0038040717696151 · Cited by 152
Greniér, P., Mourey-Gerosa, I., Benali, K., Brauner, M., Leung, A. N. C., Lenoir, S., Cordeau, M., & Mazoyer, B. M. (1996) · European Radiology · doi:10.1007/bf00181147 · Cited by 135
“To assess the intra- and interobserver variability, the kappa coefficient of agreement (K) was computed [ 8 ].”
MethodLau, J. Y. W., Sung, J. J. �., Chan, A. C. W., Lai, G. W. Y., Lau, J. T. F., Ng, E. K., Chung, S. C. S., & LI, A. K. C. (1997) · Gastrointestinal Endoscopy · doi:10.1016/s0016-5107(97)70206-2 · Cited by 135
The Working Alliance in Coaching: Why Behavior Is the Key to Success
Gessnitzer, S., & Kauffeld, S. (2015) · The Journal of Applied Behavioral Science · doi:10.1177/0021886315576407 · Cited by 181
“65 (Fleiss, 1971) across all act4consulting codes, the coders showed a substantial agreement and good interrater reliability (Fleiss, 1981; Landis & Koch, 1977).”
BackgroundA validated facial grading scale: The future of facial ageing measurement tools?
CARRUTHERS, A. J., & CARRUTHERS, J. D. A. (2010) · Journal of Cosmetic and Laser Therapy · doi:10.3109/14764172.2010.514920 · Cited by 134
Mining Software Repair Models for Reasoning on the Search Space of Automated Program Fixing
Martínez, M., & Monperrus, M. (2013) · Cited by 134
Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction
Wu, X., Zheng, W., Xia, X., & Lo, D. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3063727 · Cited by 132
“We compute the Fleiss’s Kappa Coefficient [52], [55] to measure the agreement among the three annotators of each dataset.”
MethodSensitivity and interrater agreement of CT and diffusion-weighted MR imaging in hyperacute stroke.
Saur, D., Kucinski, T., Grzyska, U., Eckert, B., Eggers, C., Niesen, W., Schoder, V., Zeumer, H., Weiller, C., & Röther, J. (2003) · PubMed Central · Cited by 134
Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System
Su, Y., Shu, L., Mansimov, E., Gupta, A., Cai, D., Lai, Y.-A., & Zhang, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.319 · Cited by 248
Spotting "hot spots" in meetings: human judgments and prosodic cues
Wrede, B., & Shriberg, E. E. (2003) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2003-747 · Cited by 152
“In [6], pairwise Kappa is modified to capture agreement among more than two raters.”
BackgroundMelia, J., Moseley, R., Ball, R. Y., Griffiths, D. F. R., Grigor, K. M., Harnden, P., Jarmulowicz, M., McWilliam, L. J., Montironi, R., Waller, M., et al. (2006) · Histopathology · doi:10.1111/j.1365-2559.2006.02393.x · Cited by 133
“Kappa statistics applicable to more than two observers and more than two categories of the variable being analysed were used.(33) An overall j was calculated for all pathologists across all scores.”
MethodMeasures of clinical agreement for nominal and categorical data: The kappa coefficient
Cyr, L., & Francis, K. T. (1992) · Computers in Biology and Medicine · doi:10.1016/0010-4825(92)90063-s · Cited by 146
Surgical inaccuracy of tumor resection and reconstruction within the pelvis: An experimental study
Cartiaux, O., Docquier, P., Paul, L., Francq, B., Cornu, O., Delloye, C., Raucent, B., Dehez, B., & Banse, X. (2008) · Acta Orthopaedica · doi:10.1080/17453670810016731 · Cited by 133
Landis, J. R., & Koch, G. G. (1975) · Statistica Neerlandica · doi:10.1111/j.1467-9574.1975.tb00254.x · Cited by 133
Hesselbrock, V. (1982) · Archives of General Psychiatry · doi:10.1001/archpsyc.1982.04290060036007 · Cited by 133
Tooth, L. R., & Ottenbacher, K. J. (2004) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2003.12.002 · Cited by 134
Irava, W. J., & Moores, K. (2010) · Journal of Family Business Strategy · doi:10.1016/j.jfbs.2010.08.002 · Cited by 172
Did It Happen? The Pragmatic Complexity of Veridicality Assessment
de Marneffe, M.-C., Manning, C. D., & Potts, C. (2012) · Computational Linguistics · doi:10.1162/coli_a_00097 · Cited by 132
“An example of committed belief is GM has laid off workers .”
BackgroundMethodInterrater reliability of the Patient Education Materials Assessment Tool (PEMAT)
Vishnevetsky, J., Walters, C. B., & Tan, K. S. (2017) · Patient Education and Counseling · doi:10.1016/j.pec.2017.09.003 · Cited by 140
“While Cohen’s kappa requires the same two raters to evaluate every material, Fleiss’ Kappa allows for the design of two raters selected from a pool of potential raters [18].”
Ad Hoc Table Retrieval using Semantic Similarity
Zhang, S., & Balog, K. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186067 · Cited by 179
Schnackenberg, A. K., Tomlinson, E., & Coen, C. (2021) · Human Relations · doi:10.1177/0018726720933317 · Cited by 127
“Fleiss’ kappa was then used as a measure of agreement among raters for the remaining items (Fleiss, 1971).”
MethodSemEval-2022 Task 5: Multimedia Automatic Misogyny Identification
Fersini, E., Gasparini, F., Rizzi, G., Saibene, A., Chulvi, B., Rosso, P., Lees, A., & Sorensen, J. S. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.74 · Cited by 273
“We estimated the inter-annotator agreement us-ing the Fleiss-κ coefficient (Fleiss, 1971).”
MethodArabic Offensive Language on Twitter: Analysis and ExperimentsPreprint
Mubarak, H., Rashed, A. A.-A., Darwish, K. M., Samih, Y., & Abdelalí, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02192 · Cited by 190
“We calculated the Inter-Annotator Agreement be-tween the annotators using Fleisss Kappa coeffi-cient (Fleiss, 1971).”
Analysis of flexible decision strategies for sustainability-focused green product recovery system
Mangla, S. K., Madaan, J., & Chan, F. T. S. (2013) · International Journal of Production Research · doi:10.1080/00207543.2013.774493 · Cited by 140
“The consistency measures are discussed and explained in detail by Fleiss (1971, 1981) and this case also uses the same measures.”
MethodA comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2
Perry, R., Whitmarsh, A., Leach, V., & Davies, P. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01819-x · Cited by 130
Annotating expressions of Appraisal in English
Read, J., & Carroll, J. (2010) · Language Resources and Evaluation · doi:10.1007/s10579-010-9135-7 · Cited by 179
Laan, G. P. M. (1997) · Speech Communication · doi:10.1016/s0167-6393(97)00012-5 · Cited by 130
“This measure was generalized to more than two observers by Fleiss (1971).”
MethodLearning the Relative Importance of Objects from Tagged Images for Retrieval and Cross-Modal Search
Hwang, S. J., & Grauman, K. L. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-011-0494-3 · Cited by 137
“Specifically, using Fleiss’ kappa statistical test for agreement between multiple raters (Fleiss 1971), we find the agreement score to be κ = 0.”
MethodPerspectives on crowdsourcing annotations for natural language processing
Wang, A., Hoang, C. D. V., & Kan, M. (2012) · Language Resources and Evaluation · doi:10.1007/s10579-012-9176-1 · Cited by 159
Prevalence of Vitiligo Among Adults in the United States
Gandhi, K., Ezzedine, K., Anastassopoulos, K. P., Patel, R., Sikirica, V., Daniel, S. R., Napatalung, L., Yamaguchi, Y., Baik, R., & Pandya, A. G. (2021) · JAMA Dermatology · doi:10.1001/jamadermatol.2021.4724 · Cited by 151
Interview Schedule for Children and Adolescents (ISCA)
Sherrill, J. T., & Kovács, M. (2000) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-200001000-00018 · Cited by 148
A Measure of Agreement for Interval or Nominal Multivariate Observations
Janson, H., & Olsson, U. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971239 · Cited by 130
“And when there is one variable and several judges, ι exactly equals Berry and Mielke’s (1988) R and also Conger’s (1980) “Fleiss exact” coefficient (κm′)—a modification of a measure originally proposed by Fleiss (1971).”
BackgroundMethodInfluentialExpiratory CT scans for chronic airway disease: correlation with pulmonary function test results.
Lucidarme, O., Coche, E., Cluzel, P., Mourey-Gerosa, I., Howarth, N., & Greniér, P. (1998) · American Journal of Roentgenology · doi:10.2214/ajr.170.2.9456933 · Cited by 130
Birko, S., Dove, E. S., & Özdemir, V. (2015) · PLOS ONE · doi:10.1371/journal.pone.0135162 · Cited by 128
The Causative Classification of Stroke system: An international reliability and optimization study
Arsava, E. M., Ballabio, E., Benner, T., Cole, J. W., Delgado-Martinez, M. P., Dichgans, M., Fazekas, F., Furie, K. L., Illoh, K., Jood, K., et al. (2010) · Neurology · doi:10.1212/wnl.0b013e3181f612ce · Cited by 128
STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits
Bhattacharya, U., Mittal, T., Chandra, R., Randhavane, T., Bera, A., & Manocha, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5490 · Cited by 160
“29, and the corresponding Fleiss’ Kappa (FK) score (Fleiss 1971) was 0.”
Papadamou, K., Papasavva, A., Zannettou, S., Blackburn, J., Kourtellis, N., Leontiadis, I., Stringhini, G., & Sirivianos, M. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7320 · Cited by 128
“After obtaining all the annotations, we compute the Fleiss agreement score ( κ ) [17] across all annotators: we find κ = 0 .”
Using Latent Class Models to Characterize and Assess Relative Error in Discrete Measurements
Espeland, M. A., & Handelman, S. L. (1989) · Biometrics · doi:10.2307/2531499 · Cited by 129
Measures of interobserver agreement: Calculation formulas and distribution effects
House, A. E. (1982) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/bf01321270 · Cited by 128
Fear conditioned with escapable and inescapable shock: Effects of a feedback stimulus.
Mineka, S., Cook, M. C., & Miller, S. (1984) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.10.3.307 · Cited by 129
Phelan, H. A., Wolf, S. E., Norwood, S. H., Aldy, K., Brakenridge, S. C., Eastman, A. L., Madden, C. J., Nakonezny, P. A., Yang, L., Chason, D. P., et al. (2012) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0b013e31825ac49e · Cited by 128
The Informational Value of Social Tagging Networks
Nam, H., & Kannan, P. K. (2014) · Journal of Marketing · doi:10.1509/jm.12.0151 · Cited by 128
“Fleiss Kappa index (Fleiss 1971) for the reliability across three raters was .904 (z = 44.7, p .001), indicating a reasonable level of agreement.”
BackgroundInfluentialLatent class analysis of diagnostic agreement
Uebersax, J. S., & Grove, W. M. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090509 · Cited by 128
Buchbinder, R., Goel, V., Bombardier, C., & Hogg‐Johnson, S. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00519-6 · Cited by 143
Hopkins, M., Spillane, J. P., Jakopovic, P. M., & Heaton, R. M. (2013) · The Elementary School Journal · doi:10.1086/671935 · Cited by 160
“Once kappa coefficients (Fleiss, 1971 , 1981 ) of .”
BackgroundBlasco, J., Aleixos, N., Gómez‐Sanchís, J., & Moltó, E. (2009) · Biosystems Engineering · doi:10.1016/j.biosystemseng.2009.03.009 · Cited by 131
Social commerce—state-of-the-art and future research directions
Baethge, C., Klier, J., & Klier, M. (2016) · Electronic Markets · doi:10.1007/s12525-016-0225-2 · Cited by 158
“We observed a value for Fleiss’ Kappa of 87.3 %.”
MethodInfluentialWhy do people play location-based augmented reality games: A study on Pokémon GO
Alha, K., Koskinen, E., Paavilainen, J., & Hamari, J. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.008 · Cited by 158
Carlson, D. A., Ravi, K., Kahrilas, P. J., Gyawali, C. P., Bredenoord, A. J., Castell, D. O., Spechler, S. J., Halland, M., Kanuri, N. D., Katzka, D. A., et al. (2015) · The American Journal of Gastroenterology · doi:10.1038/ajg.2015.159 · Cited by 126
“Classic Achalasia (11) ADLESR (17) DES (2) Nutcracker (2) Nutcracker+HTN LES (0) Isolated HTN LES (1) IEM (3) Normal (4) Total (40)”
BackgroundInfluentialBirk, S., Willby, N. J., KELLY, M. G., Bonne, W., Borja, Á., Poikāne, S., & van de Bund, W. (2013) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2013.03.037 · Cited by 125
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126
Emotions: The Unexplored Fuel of Fake News on Social Media
Horner, C. G., Galletta, D. F., Crawford, J. C., & Shirsat, A. R. (2021) · Journal of Management Information Systems · doi:10.1080/07421222.2021.1990610 · Cited by 191
“91 respectively, indicating excellent agreement [24][55].”
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.