Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 701–800 of 1,118
Listener expertise and sound identification influence the categorization of environmental sounds.
Lemaître, G., Houix, O., Misdariis, N., & Susini, P. (2010) · Journal of Experimental Psychology Applied · doi:10.1037/a0018762 · Cited by 112
“48 for the nonexperts’ categories (Fleiss, 1971).”
BackgroundInfluentialLearning how to Differ: Agreement and Reliability Statistics in Psychiatry
David, L. S. (1995) · The Canadian Journal of Psychiatry · doi:10.1177/070674379504000202 · Cited by 97
“The first problem, having more than two judges, has been dealt with by a relatively simple extension of the equation (10,11).”
BackgroundFisher, S. L., Bucholz, K. K., Reich, W., Fox, L., Kuperman, S., Kramer, J., Hesselbrock, V., Dick, D. M., Nurnberger, J. I., Edenberg, H. J., et al. (2006) · Alcoholism: Clinical and Experimental Research · doi:10.1111/j.1530-0277.2006.00205.x · Cited by 96
“k coefficients (Fleiss, 1971) were also computed to allow for cross-study comparison.”
MethodUsing affective parameters in a content-based recommender system for images
Tkalčič, M., Burnik, U., & Košir, A. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9079-z · Cited by 108
“For the given ten genres, three annotators and 70 items the Fleiss’ kappa inter rater agreement measure (Fleiss 1971) was κ = 0 .”
BackgroundKehrer, C., Blumenstock, G., Raabe, C., & Krägeloh‐Mann, I. (2010) · Developmental Medicine & Child Neurology · doi:10.1111/j.1469-8749.2010.03821.x · Cited by 96
Search-based software library recommendation using multi-objective optimization
Ouni, A., Kula, R. G., Kessentini, M., Ishio, T., Germán, D. M., & Inoue, K. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.11.007 · Cited by 117
Deep learning for detecting inappropriate content in text
Yenala, H., Jhanwar, A., Chinnakotla, M. K., & Goyal, J. (2017) · International Journal of Data Science and Analytics · doi:10.1007/s41060-017-0088-4 · Cited by 115
“The inter-annotator agreement based on fleiss’ kappa value [9] was found to be 0.”
MethodBeeckman, D., den Bussche, K. V., Alves, P. J. P., Long, M. C. A., Beele, H., Ciprandi, G., Coyer, F. M., de Groot, T., Meyer, D. D., Deschepper, E., et al. (2018) · British Journal of Dermatology · doi:10.1111/bjd.16327 · Cited by 117
Understanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories
Mohammad, S. M., & Kiritchenko, S. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5or8sy3gqobf · Cited by 99
“Fleiss’ κ, on the other hand, calculates the extent to which the observed agreement exceeds the one that would be expected by chance (Fleiss, 1971).”
BackgroundHow Practitioners Perceive Automated Bug Report Management Techniques
Zou, W., Lo, D., Chen, Z., Xia, X., Feng, Y., & Xu, B. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2870414 · Cited by 96
“Following [24], [19], Fleiss Kappa [23] was used to measure the agreement between the two label-ers, and the overall Kappa value was 0.93.”
MethodInfluentialPatey, A. M., Grimshaw, J., & Francis, J. (2021) · Implementation Science · doi:10.1186/s13012-021-01089-0 · Cited by 101
The clinical grading of lens opacities
Taylor, H. R., & West, S. K. (1989) · Australian and New Zealand Journal of Ophthalmology · doi:10.1111/j.1442-9071.1989.tb00492.x · Cited by 96
Evaluation of the Turkish translation of a disease activity form for Behcet's syndrome
Hamuryudan, V., Fresko, İ., Di̇reskeneli̇, H., Tenant, M. J., Yurdakul, S., Akoğlu, T., & Yazıcı, H. (1999) · Lara D. Veeken · doi:10.1093/rheumatology/38.8.734 · Cited by 95
Naïve listeners' prominence and boundary perception
Mo, Y., Cole, J., & Lee, E. K. (2008) · Speech prosody · doi:10.21437/speechprosody.2008-163 · Cited by 95
“The reliability of the naive prosody transcriptions pooled over all 74 transcribers is evaluated using Fleiss’ kappa coefficient [ 2 ] and their z-normalized scores, as shown in Table 1. We choose Fleiss’ kappa coefficient because it provides a single coefficient as a measure of agreement between all pairs of transcri…”
Fleischmann-Struzek, C., Thomas-Rüddel, D. O., Schettler, A., Schwarzkopf, D., Stacke, A., Seymour, C. W., Haas, C., Dennler, U., & Reinhart, K. (2018) · PLOS ONE · doi:10.1371/journal.pone.0198847 · Cited by 95
Medical Diagnostic Tests: A Review of Test Anatomy, Phases, and Statistical Treatment of Data
Bolboaca, S. D. (2019) · Computational and Mathematical Methods in Medicine · doi:10.1155/2019/1891569 · Cited by 108
“(i) Nominal or ordinal scale: Cohen’s kappa coefficient (nominal scale, inclusive dichotomial such as positive/negative test result), symbol κ [128], and its derivatives (Fleiss’ generalized kappa [129], Conger’s generalized kappa [130], and weighted kappa (ordinal scale test result)) [131]”
BackgroundRuehland, W. R., O’Donoghue, F. J., Pierce, R. J., Thornton, A. T., Singh, P., Copland, J. M., Stevens, B., & Rochford, P. D. (2011) · SLEEP · doi:10.1093/sleep/34.1.73 · Cited by 106
Herjanic, B. M., & Reich, W. (1997) · Research on Child and Adolescent Psychopathology · doi:10.1023/a:1025703323438 · Cited by 95
Category Distinguishability and Observer Agreement
Darroch, J. N., & McCloud, P. (1986) · Australian Journal of Statistics · doi:10.1111/j.1467-842x.1986.tb00709.x · Cited by 99
Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction
Tan, Q., Xu, L., Bing, L., Ng, H. T., & Aljunied, S. M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.580 · Cited by 207
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
InfluentialSources of variation in emotional awareness: Age, gender, and socioeconomic status
Mankus, A. M., Boden, M. T., & Thompson, R. J. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2015.09.043 · Cited by 93
Neural personalized response generation as domain adaptation
Zhang, W.-N., Zhu, Q., Wang, Y., Zhao, Y., & Liu, T. (2019) · World Wide Web · doi:10.1007/s11280-018-0598-6 · Cited by 94
“The agreement is calculated by using the Fleiss’ kappa [11] Values in bold denote the best performance on each corresponding evaluation metrics”
MethodCulture, Styles of Institutional Interactions, and Inequalities in Healthcare Experiences
Gage‐Bouchard, E. A. (2017) · Journal of Health and Social Behavior · doi:10.1177/0022146517693051 · Cited by 105
Detecting similar repositories on GitHub
Zhang, Y., Lo, D., Kochhar, P. S., Xia, X., Li, Q., & Sun, J. (2017) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2017.7884605 · Cited by 93
“We used Fleiss Kappa [13] to evaluate the agreement between the two annotators.”
MethodArichi, T., Whitehead, K., Barone, G., Pressler, R., Padormo, F., Edwards, A. D., & Fabrizi, L. (2017) · eLife · doi:10.7554/elife.27814 · Cited by 93
“Inter-rater reliability was assessed using Fleiss’ Kappa analysis (Fleiss, 1971) and resulted in a substantial agreement (median Fleiss’ Kappa 0.65 (range 0.25–0.76)).”
MethodInfluentialKennedy, K. M., Zilkens, R., Allen, W. M., Foo, K. Y., Fang, Q., Chin, L., Sanderson, R. W., Anstie, J. D., Wijesinghe, P., Curatolo, A., et al. (2020) · Cancer Research · doi:10.1158/0008-5472.can-19-1240 · Cited by 97
“Interreader agreement was quantified using the Fleiss' Kappa metric (51).”
MethodHate and offensive speech detection on Arabic social media
Alsafari, S., Sadaoui, S., & Mouhoub, M. (2020) · Online Social Networks and Media · doi:10.1016/j.osnem.2020.100096 · Cited by 115
Dahlke, J., Bogner, K., Becker, M., Schlaile, M. P., Pyka, A., & Ebersberger, B. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120799 · Cited by 115
“We calculate the reliability between coders using Fleiss’ kappa, which constitutes an appropriate measure considering that there are more than two coders and that categories are assigned based on latent characteristics, which are neither quantifiable and weighted nor ordered (Fleiss, 1971).”
MethodPLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning
Bao, S., He, H., Wang, F., Wu, H., Wang, H., Wu, W., Guo, Z., Liu, Z., & Xu, X. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.222 · Cited by 94
Sales and operations planning and the firm performance
Thomé, A. M. T., Scavarda, L. F., Fernandez, N. S., & Scavarda, A. (2012) · International Journal of Productivity and Performance Management · doi:10.1108/17410401211212643 · Cited by 102
“The agreement measured by Cohen’s kappa for three judges on abstract reviews was 0.47 with a standard deviation of 0.12, which was significantly different from agreement by chance alone (Fleiss, 1971).”
BackgroundMethodInfluentialMüller, C. M., de Vos, R. A. I., Maurage, C.-A., Thal, D. R., Tolnay, M., & Braak, H. (2005) · Journal of Neuropathology & Experimental Neurology · doi:10.1097/01.jnen.0000171652.40083.15 · Cited by 95
“The high overall agreement of all raters is reflected in the statistical significance calculated from the logistic regression model proposed by Fleiss (10).”
MethodPrevalence of personality disorders among combat veterans with posttraumatic stress disorder
Bollinger, A. R., Riggs, D. S., Blake, D. D., & Ruzek, J. I. (2000) · Journal of Traumatic Stress · doi:10.1023/a:1007706727869 · Cited by 98
“Interrater reliability for the SCID-II was assessed using kappa coefficients for multiple raters ( Fleiss, 1971 ).”
MethodMak, A., Cheak, A. A., Tan, J. Y. S., Su, H., Ho, R., & Lau, C. (2009) · Rheumatology · doi:10.1093/rheumatology/kep120 · Cited by 99
Organizational culture change: An inter‐group attributional analysis
Silvester, J., Anderson, N. R., & Patterson, F. (1999) · Journal of Occupational and Organizational Psychology · doi:10.1348/096317999166464 · Cited by 96
Traceability in the wild: automatically augmenting incomplete trace links
Rath, M., Rendall, J. B., Guo, J., Cleland‐Huang, J., & Mäder, P. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180207 · Cited by 132
Jacobson, N. S., & Anderson, E. A. (1982) · Behavior Therapy · doi:10.1016/s0005-7894(82)80037-3 · Cited by 94
Comparing the Joint Agreement of Several Raters with Another Rater
Williams, G. W. (1976) · Biometrics · doi:10.2307/2529750 · Cited by 94
Methodological Principles for Reproducible Performance Evaluation in Cloud Computing
Papadopoulos, A. V., Versluis, L., Bauer, A., Herbst, N., Kistowski, J. V., Ali-Eldin, A., Abad, C. L., Amaral, J. N., Tuma, P., & Iosup, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2927908 · Cited by 93
“We use the Fleiss’ Kappa analysis [66], [67] and the weighted Cohen’s Kappa analysis [68] to measure the degree of agreement between reviewers who rate a sample of a cohort; in our work, the cohort is the complete set of papers in the reviewing process.”
MethodUnsupervised Topic Modeling for Short Texts Using Distributed Representations of Words
Sridhar, V. K. R. (2015) · Workshop on Vector Space Modeling for Natural Language Processing (VSM-NLP) · doi:10.3115/v1/w15-1526 · Cited by 97
“We computed the inter annotator agreement be-tween the three labelers using Fleiss’ kappa metric (Fleiss, 1971).”
MethodAstridge, B., Li, W. W., McDERMOTT, B. M., & Longhitano, C. (2023) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2023.106055 · Cited by 126
Bangdiwala, S. I., & Shankar, V. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-97 · Cited by 104
Systematic reviews to evaluate diagnostic tests
Khan, K. S., Dinnes, J., & Kleijnen, J. (2001) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/s0301-2115(00)00463-2 · Cited by 108
Leukoaraiosis, intracerebral hemorrhage, and arterial hypertension.
Inzitari, D., Giordano, G. P., Ancona, A. L., Pracucci, G., Mascalchi, M., & Amaducci, L. (1990) · Stroke · doi:10.1161/01.str.21.10.1419 · Cited by 93
“They presented with transient ischemic attack or ischemic stroke (31), trauma (28), cerebral neoplasms (28), nonvascular or vascular dementia (18), epilepsy (11), headache (10), vertigo or syncope (8), depression (5), subarachnoid hemorrhage (5), deafness (2), trigeminal neuralgia (2), or miscellaneous complaints (7).”
BackgroundTracking: Prediction of Future Values from Serial Measurements
Ware, J. H., & Wu, M. C. (1981) · Biometrics · doi:10.2307/2530556 · Cited by 93
Connections Between the Design Tool, Design Attributes, and User Preferences in Early Stage Design
Häggman, A., Tsai, G., Elsen, C., Honda, T., & Yang, M. C. (2015) · Journal of Mechanical Design · doi:10.1115/1.4030181 · Cited by 96
“Fleiss’ Kappa was chosen as the inter-rater reliability metric because it allows more than two raters [49].”
MethodUsability research in educational technology: a state-of-the-art systematic review
Lu, J., Schmidt, M., Lee, M., & Huang, R. (2022) · Educational Technology Research and Development · doi:10.1007/s11423-022-10152-6 · Cited by 143
AbdulSabur, N. Y., Xu, Y., Liu, S., Chow, H. M., Baxter, M., Carson, J., & Braun, A. R. (2014) · Cortex · doi:10.1016/j.cortex.2014.01.017 · Cited by 110
“Inter-rater reliability was assessed by Fleiss’ kappa statistics (Fleiss, 1971), which yielded a value (kappa ¼ .9696) indicating perfect agreement between the raters (Landis & Koch, 1977).”
MethodLogan, K. J., & Conture, E. G. (1997) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jslhr.4001.107 · Cited by 92
Shin, S., King, C. S., Brown, A. W., Albano, M. C., Atkins, M. B., Sheridan, M. J., Ahmad, S., Newton, K., Weir, N. A., Shlobin, O. A., et al. (2014) · Respiratory Medicine · doi:10.1016/j.rmed.2014.08.009 · Cited by 94
“The intraclass correlation coefficient was used to assess the level of agreement among readers for the continuous variables of PA diameter and PA:A ratio [19,20].”
MethodAssortative mating among in-patients with primary affective disorder
Merikangas, K. R., & Spiker, D. G. (1982) · Psychological Medicine · doi:10.1017/s0033291700049059 · Cited by 93
Time-Varying EEG Correlations Improve Automated Neonatal Seizure Detection
Tapani, K., Vanhatalo, S., & Stevenson, N. J. (2018) · International Journal of Neural Systems · doi:10.1142/s0129065718500302 · Cited by 91
“The complexity of translation between cortical event and EEG waveform precludes accurate explanation, however, clinical studies support the idea that a wide range of spatio-temporal seizure configurations should be targeted by an ideal SDA.37,38 The proposed SDA significantly outperformed our implementations of two st…”
MethodInfluentialOffensive Comments in the Brazilian Web: a dataset and baseline results
de Pelle, R. P., & Moreira, V. P. (2017) · Anais do Brazilian Workshop on Social Network Analysis and Mining (BraSNAM) · doi:10.5753/brasnam.2017.3260 · Cited by 120
Assessing Inter-Annotator Agreement for Medical Image Segmentation
Yang, F., Zamzmi, G., Angara, S., Rajaraman, S., Aquilina, A., Xue, Z., Jaeger, S., Papagiannakis, E., & Antani, S. (2023) · IEEE Access · doi:10.1109/access.2023.3249759 · Cited by 119
“Although the kappa coefficient [7], [41] has been widely used to assess inter-annotator agreement, most previous works explored it with classification tasks such as cervical cancer classification [25], [26], [27] and disease/survival prediction in CXR [29], [30], [31], among others.”
BackgroundMethodvon Arx, T., Hänni, S., & Jensen, S. S. (2010) · Journal of Endodontics · doi:10.1016/j.joen.2010.03.040 · Cited by 94
Interrater Agreement and Reliability
Goodwin, L. D. (2001) · Measurement in Physical Education and Exercise Science · doi:10.1207/s15327841mpee0501_2 · Cited by 96
“sides percentage of agreement that can be used to determine the extent of rater or observer agreement; see Frick and Semmel (1978) for a comprehensive review. Also, Looney (1989) described and illustrated proportion of agreement and related statistics that can be used to estimate reliability of criterion-referenced me…”
BackgroundInfluentialNielsen, P. S., Lindebjerg, J., Rasmussen, J., Starklint, H., Waldstrøm, M., & Nielsen, B. (2010) · Human Pathology · doi:10.1016/j.humpath.2010.05.015 · Cited by 98
“The κ estimates describing interobserver agreement were based on all the pathologists reviewing the same slides once using the same method [11].”
MethodIncidental emotions in moral dilemmas: The influence of emotion regulation
Szekely‐Copîndean, R. D., & Miu, A. C. (2014) · Cognition & Emotion · doi:10.1080/02699931.2014.895300 · Cited by 113
“An inter-rater agreement was considered excellent if Cohen’s k was over 0.75, fair to good if it ranged between 0.4 and 0.75 and poor if it was below 0.4 (Fleiss, 1971).”
MethodInfluentialKvasnicka, H. M., Beham‐Schmid, C., Bob, R., Dirnhofer, S., Hussein, K., Kreipe, H. H., Kremer, M., Schmitt-Graeff, A. H., Schwarz, S., Thiele, J., et al. (2015) · Histopathology · doi:10.1111/his.12871 · Cited by 99
“Fleiss kappa statistics were used to determine the interobserver agreement on the final grading.(24) Kappa statistics measure the observed level of agreement between panel members for a set of ratings and correct the result for agreement that would be expected by chance.”
MethodSorrel, M. Á., Olea, J., Abad, F. J., TORRE, J. D. L., Aguado, D., & Lievens, F. (2016) · Organizational Research Methods · doi:10.1177/1094428116630065 · Cited by 115
“As done in Li and Suen (2013), we computed the Fleiss’s kappa statistic (Fleiss, 1971) to evaluate the interrater reliability of the judgments made.”
MethodWhat Do We Teach When We Teach the Learning Sciences? A Document Analysis of 75 Graduate Programs
Sommerhoff, D., Szameitat, A., Vogel, F., Chernikova, O., Stockinger, K., & Fischer, F. (2018) · Journal of the Learning Sciences · doi:10.1080/10508406.2018.1440353 · Cited by 102
“For this, Fleiss’ kappa (Fleiss, 1971; Fleiss & Cohen, 1973) was selected, which is usually used to determine interrater reliability but can also be used to determine the similarity of “objects” other than raters.”
MethodEtminani, K., Soliman, A., Davidsson, A., Chang, J. R., Martínez‐Sanchis, B., Byttner, S., Camacho, V., Bauckneht, M., Stegeran, R., Ressner, M., et al. (2021) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-021-05483-0 · Cited by 101
The effect of self-assessment on academic performance and the role of explicitness: a meta-analysis
Yan, Z., Xiang, W., Boud, D. J., & Lao, H. (2021) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2021.2012644 · Cited by 183
“Inter-rater reliability ( kappa ) between the two coders was .81, above the minimum recommendation (Fleiss 1971).”
BackgroundKennedy, B., Atari, M., Davani, A. M., Yeh, L., Omrani, A., Kim, Y., Coombs, K., Havaldar, S., Portillo-Wightman, G., Gonzalez, E., et al. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-021-09569-x · Cited by 136
“We downloaded Gab posts from the public dump of data by Pushshift.io4 (Gaffney, 2018).”
MethodMoharreri, K., Ha, M., & Nehm, R. H. (2014) · Evolution Education and Outreach · doi:10.1186/s12052-014-0015-2 · Cited by 90
“Although different studies have suggested different cutoff scores for Cohen ’ s kappa, the benchmarks suggested by Landis and Koch (1977) and Fleiss (1971) were used in the evaluation of EvoGrader performance: ‘ almost perfect ’ : 0.81-1.00 (Landis and Koch 1977) and ‘ excellent ’ : 0.75 – 1.00…”
BackgroundMethodInfluentialStulberg Classification System for Evaluation of Legg-Calvé-Perthes Disease
Neyt, J., Weinstein, S. L., Spratt, K. F., Dolan, L. A., Morcuende, J. A., Dietz, F. R., Guyton, G. P., Hart, R. A., Kraut, M. A. S., Lervick, G. N., et al. (1999) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-199909000-00002 · Cited by 90
Permutation tests for joinpoint regression with applications to cancer rates
Kim, H.-J., Fay, M. P., Feuer, E. J., & Midthune, D. N. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000215)19:3<335::aid-sim336>3.0.co;2-z · Cited by 6,318
Cohen, D. A., Dabbs, D. J., Cooper, K. L., Amin, M., Jones, T. E., Jones, M. W., Chivukula, M., Trucco, G., & Bhargava, R. (2012) · American Journal of Clinical Pathology · doi:10.1309/ajcp6dkrnd5ckvdd · Cited by 92
Aid is Out There: Looking for Help from Tweets during a Large Scale Disaster
Varga, I., Sano, M., Torisawa, K., Hashimoto, C., Ohtake, K., Kawai, T., Oh, J., & Saeger, S. D. (2013) · Cited by 90
Heenan, P. J., Matz, L. R., Blackwell, J. B., Kelsall, G. R. H., SINGH, A., Seldam, R. E. J. T., & Holman, C. (1984) · Histopathology · doi:10.1111/j.1365-2559.1984.tb02388.x · Cited by 90
The effect of intrauterine growth retardation on the quality of general movements in the human fetus
Sival, D. A., Visser, G. H. A., & Prechtl, H. F. R. (1992) · Early Human Development · doi:10.1016/0378-3782(92)90107-r · Cited by 90
Technology business incubators and industrial development: resource‐based view
Somsuk, N., Wonglimpiyarat, J., & Laosirihongthong, T. (2012) · Industrial Management & Data Systems · doi:10.1108/02635571211204281 · Cited by 135
Measuring opportunity evaluation: Conceptual synthesis and scale development
Scheaf, D. J., Loignon, A. C., Webb, J. W., Heggestad, E. D., & Wood, M. S. (2019) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2019.04.003 · Cited by 118
Meta-Learning for Low-resource Natural Language Generation in Task-oriented Dialogue Systems
Mi, F., Huang, M., Zhang, J., & Faltings, B. V. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/437 · Cited by 116
Grimmett, L., Whitsed, R., & Horta, A. (2020) · Ecological Modelling · doi:10.1016/j.ecolmodel.2020.109194 · Cited by 104
The Influence of Clinical Information in the Histopathologic Diagnosis of Melanocytic Skin Neoplasms
Ferrara, G., Argényi, Z. B., Argenziano, G., Cerio, R., Cerroni, L., Blasi, A. D., Feudale, E. A. A., Giorgio, C. M., Massone, C., Nappi, O., et al. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005375 · Cited by 90
“The k statistics for multiple observers and the standard errors for were calculated by using the method reported by Fleiss [23–25]: k values range between +1 (perfect agreement) and 21 (perfect disagreement); values greater than 0.”
MethodGrayson, K. A., & Rust, R. T. (2001) · Journal of Consumer Psychology · doi:10.1207/s15327663jcp1001&2_06 · Cited by 127
On Agreement Indices for Nominal Data
Popping, R. (1988) · Sociometric Research · doi:10.1007/978-1-349-19051-5_6 · Cited by 88
Accuracy of High-Speed Video Analysis to Diagnose Primary Ciliary Dyskinesia
Rubbo, B., Shoemark, A., Jackson, C. L., Hirst, R. A., Thompson, J., Hayes, J., Frost, E., Copeland, F., Hogg, C. L., O’Callaghan, C., et al. (2019) · CHEST Journal · doi:10.1016/j.chest.2019.01.036 · Cited by 105
“We calculated the interobserver repeatability using the Fleiss k coefficient for each diagnostic outcome.(16) We calculated the intraobserver repeatability for each of the scientists using the Cohen k coefficient, with bootstrapped confidence intervals (n 1⁄4 5).”
MethodVirtual Assistance in Any Context: A Taxonomy of Design Elements for Domain-Specific Chatbots
Janssen, A., Passlick, J., Cardona, D. R., & Breitner, M. H. (2020) · Business & Information Systems Engineering · doi:10.1007/s12599-020-00644-1 · Cited by 117
“As a result, we were able to calculate the quality of the agreement with the Fleiss’ (1971) kappa coefficient which is 0.63.”
MethodInfluentialWhat is the Best Clinical Test for Assessment of the Teres Minor in Massive Rotator Cuff Tears?
Collin, P., Treseder, T., Denard, P. J., Neyton, L., Walch, G., & Lädermann, A. (2015) · Clinical Orthopaedics and Related Research · doi:10.1007/s11999-015-4392-9 · Cited by 88
“The interobserver reliability of the Goutallier scores were determined for each set of images by calculating a multijudge j coefficient of agreement [6].”
Yang, J., Yao, S., Zhu, X., Zhang, C., Ling, Y., Abela, J. R. Z., Esseling, P. G., & McWhinnie, C. M. (2010) · Journal of Affective Disorders · doi:10.1016/j.jad.2010.04.023 · Cited by 95
“The Fleiss kappa was .73 (Fleiss, 1971). Criteria proposed by Landis and Koch (1977) were used to interpret the kappa coefficients: excellent, kappaN .”
MethodThe Relationship of Lumbar Flexion to Disability in Patients With Low Back Pain
Sullivan, M. S., Shoaf, L. D., & Riddle, D. L. (2000) · Physical Therapy · doi:10.1093/ptj/80.3.240 · Cited by 88
Additional Injuries in Young Infants with Concern for Abuse and Apparently Isolated Bruises
Harper, N. S., Feldman, K. W., Sugar, N. F., Anderst, J. D., & Lindberg, D. M. (2014) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2014.04.004 · Cited by 94
Topol, G. A., Podestá, L. A., Reeves, K. D., Giraldo, M. M., Johnson, L. L., Grasso, R., Jamín, A., Clark, T. J., & Rabago, D. (2016) · PM&R · doi:10.1016/j.pmrj.2016.03.008 · Cited by 88
“The results of section-by-section arthroscopic video analysis of each of 9 sections of the medial condyle in all 6 participants by each surgeon-reader were summarized for display on a medial condyle map, and compared for inter-reader reliability using a Fleiss k statistic [24].”
MethodNominal Scale Agreement Among Observers
Schouten, H. J. A. (1986) · Psychometrika · doi:10.1007/bf02294066 · Cited by 90
JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs
Ke, P., Ji, H., Ran, Y., Cui, X., Wang, L., Song, L., Zhu, X., & Huang, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.223 · Cited by 92
Wilson, M. J., & Wilson, M. L. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22758 · Cited by 107
“Our Fleiss Kappa scores are reported inline below as we describe the scales we produced.”
MethodAnalysing Intraclass Correlation for Dichotomous Variables
Mak, T. K. (1988) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2347309 · Cited by 87
Tools for Automated Analysis of Cybercriminal Markets
Portnoff, R. S., Afroz, S., Durrett, G., Kummerfeld, J. K., Berg-Kirkpatrick, T., McCoy, D., Levchenko, K., & Paxson, V. (2017) · The Web Conference · doi:10.1145/3038912.3052600 · Cited by 104
“…Forums training set (728 posts, 3 annotators per post, 30 posts with 8 annotators per post) • Hack Forums test set (140 posts, 4 annotators per post) We used the Fleiss Kappa measurement of inter-annotator agreement [Fleiss 1971] and found that our annotations had “substantial agreement” ( κ = 0 .”
MethodPossession vs. Direct Play: Evaluating Tactical Behavior in Elite Soccer
Kempe, M., Vogelbein, M., Memmert, D., & Nopp, S. (2014) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.5923/s.sports.201401.05 · Cited by 87
“between the five raters [25].”
A Randomized Clinical Trial of the Leboyer Approach to Childbirth
Nelson, N. M., Enkin, M. W., Saigal, S., BENNETT, K. J., MILNER, R. A., & Sackett, D. L. (1980) · New England Journal of Medicine · doi:10.1056/nejm198003203021203 · Cited by 87
Crowdsourcing a Word-Emotion Association LexiconPreprint
Mohammad, S. M., & Turney, P. D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1308.6297 · Cited by 87
“However, it is appropriate only when the same judges annotate all the instances (Fleiss 1971).”
BackgroundInfluentialArchitecture of the World Wide Web
Halpin, H. (2012) · Semantic web and beyond · doi:10.1007/978-1-4614-1885-6_2 · Cited by 89
The Generic Integrity of Newspaper Editorials
Ansary, H., & Babaii, E. (2005) · RELC Journal · doi:10.1177/0033688205060051 · Cited by 86
Accurate Language Identification of Twitter Messages
Lui, M., & Baldwin, T. (2014) · doi:10.3115/v1/w14-1303 · Cited by 95
“Table 1: Inter-annotator agreement measured using Fleiss’ kappa (Fleiss, 1971) over annotations for TWITTER.”
MethodA matter of perspective – the role of interpersonal relationships in supply chain risk management
Durach, C. F., & Machuca, J. A. D. (2018) · International Journal of Operations & Production Management · doi:10.1108/ijopm-03-2017-0157 · Cited by 109
Mahmoudi, A., Abbasi, M., & Deng, X. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116067 · Cited by 105
“To overcome this barrier, Fleiss’ kappa can be utilized (Fleiss, 1971), which is suitable for calculating the reliability of agreement (consistency) among a group of scenarios as well as for ordinal and nominal data.”
MethodCollocation and Thai Word Segmentation
Aroonmanakun, W. (2002) · Cited by 86
Influence of contextual information in emotion annotation for spoken dialogue systems
Callejas, Z., & López-Cózar, R. (2008) · Speech Communication · doi:10.1016/j.specom.2008.01.001 · Cited by 86
“The simplest Kappa coefficient used was proposed by Fleiss (1971), as a generalization for multiple annotators of the two-coders Scott's p (Scott, 1955).”
MethodEnabling knowledge creation through outsiders: towards a push model of open innovation
Spaeth, S., Stuermer, M., & von Krogh, G. (2010) · International Journal of Technology Management · doi:10.1504/ijtm.2010.035983 · Cited by 103
“Two authors coded the randomly selected messages independently and inter-rater reliability was measured using Fleiss Kappa (Fleiss, 1971).”
Method
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.