Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 301–400 of 1,118
Riskind, J. H. (1987) · Archives of General Psychiatry · doi:10.1001/archpsyc.1987.01800210065010 · Cited by 184
Chen, G., Faris, P., Hemmelgarn, B., Walker, R. L., & Quan, H. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-5 · Cited by 183
Fisher, C. G., Schouten, R., Versteeg, A. L., Boriani, S., Varga, P. P., Rhines, L. D., Kawahara, N., Fourney, D., Weir, L., Reynolds, J. J., et al. (2014) · Radiation Oncology · doi:10.1186/1748-717x-9-69 · Cited by 183
“Inter-observer agreement was calculated for both the radiation oncologists and the surgeons based on the first review using the Fleiss’ kappa for multiple raters [9].”
Macroalgal blooms alter community structure and primary productivity in marine ecosystems
Lyons, D. A., Arvanitidis, C., Blight, A. J., Chatzinikolaou, E., Guy‐Haim, T., Kotta, J., Orav‐Kotta, H., Queirós, A. M., Rilov, G., Somerfield, P. J., et al. (2014) · Global Change Biology · doi:10.1111/gcb.12644 · Cited by 183
“We assessed consistency among reviewers with Fleiss’ Kappa statistic (Fleiss, 1971), using a Kappa of at least 0.”
MethodInfluentialBrowne, G. B., Arpin, K., Corey, P., Fitch, M., & Gafni, A. (1990) · Medical Care · doi:10.1097/00005650-199001000-00006 · Cited by 183
Assessing the Reliability and Validity of Expert Interviews
Dorussen, H., Lenz, H., & Blavoukos, S. (2005) · European Union Politics · doi:10.1177/1465116505054835 · Cited by 182
Measures of interobserver agreement: Calculation formulas and distribution effects
House, A. E., House, B. J., & Campbell, M. B. (1981) · Journal of Behavioral Assessment · doi:10.1007/bf01321350 · Cited by 180
High-density surface EMG maps from upper-arm and forearm muscles
Rojas-Martínez, M., Mañanas, M. A., & Alonso, J. F. (2012) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-9-85 · Cited by 181
“Fleiss’ Kappa index [24] was used to measure agreement between experts, scoring 82.”
MethodBrennan, P. F., & Hays, B. J. (1992) · Research in Nursing & Health · doi:10.1002/nur.4770150210 · Cited by 180
Repeatability and Reproducibility of Decisions by Latent Fingerprint Examiners
Ulery, B. T., Hicklin, R. A., Buscaglia, J., & Roberts, M. A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032800 · Cited by 180
“Fleiss’s kappa [12] corrects the pure, descriptive metric to account for agreements due to chance.”
BackgroundSchwendicke, F., Elhennawy, K., Paris, S., Friebertshäuser, P., & Krois, J. (2020) · Journal of Dentistry · doi:10.1016/j.jdent.2019.103260 · Cited by 180
Infusion phlebitis assessment measures: a systematic review
Ray‐Barruel, G., Polit, D. F., Murfield, J. E., & Rickard, C. M. (2014) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12107 · Cited by 179
“the design was not fully crossed), it is unclear whether the appropriate statistic – Fleiss’s kappa [78,79] rather than Cohen’s kappa [80] – was used.”
Antidepressant Use during Pregnancy and the Rates of Spontaneous Abortions: A Meta-Analysis
Hemels, M. E., Einarson, A., Koren, G., Lanctôt, K. L., & Einarson, T. R. (2005) · Annals of Pharmacotherapy · doi:10.1345/aph.1e547 · Cited by 179
Peters, L., & Andrews, G. (1995) · Psychological Medicine · doi:10.1017/s0033291700033237 · Cited by 179
Videofluorographic Study of Swallowing in Parkinson's Disease
Nagaya, M., Kachi, T., Yamada, T., & Igata, A. (1998) · Dysphagia · doi:10.1007/pl00009562 · Cited by 179
Automatic Summarization of Bug Reports
Rastkar, S., Murphy, G. C., & Murray, G. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.2297712 · Cited by 178
“sure the level of agreement amongst the annotators with regards to bug report sentences that they linked in their abstractive summaries [29].”
BackgroundHoang, J. K., Middleton, W. D., Farjat, A. E., Teefey, S. A., Abinanti, N., Boschini, F. J., Bronner, A. J., Dahiya, N., Hertzberg, B. S., Newman, J. R., et al. (2018) · American Journal of Roentgenology · doi:10.2214/ajr.17.19192 · Cited by 178
Interrater reliability of sleep stage scoring: a meta-analysis
Lee, Y. J., Lee, J. Y., Cho, J. H., & Choi, J. H. (2022) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.9538 · Cited by 174
Moral foundations twitter corpus: a collection of 35k tweets annotated for moral sentiment
Hoover, J., Portillo-Wightman, G., Yeh, L., Havaldar, S., Davani, A. M., Lin, Y., Kennedy, B., Atari, M., Kamel, Z., Mendlen, M., et al. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619876629 · Cited by 177
“To evaluate interannotator agreement, we calculated both Fleiss’s (1971) kappa and prevalence-and bias-adjusted Fleiss’s kappa (PABAK; Sim & Wright, 2005) for multiple annotators (See Table 6).”
MethodDevelopment of the responsiveness to child feeding cues scale
Hodges, E. A., Johnson, S. L., Hughes, S. O., Hopkinson, J. M., Butte, N. F., & Fisher, J. O. (2013) · Appetite · doi:10.1016/j.appet.2013.02.010 · Cited by 177
Weakly-Supervised Deep Embedding for Product Review Sentiment Analysis
Zhao, W., Guan, Z., Chen, L., He, X., Cai, D., Wang, B., & Wang, Q. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2756658 · Cited by 177
“We used Fleiss’s kappa [13] to measure”
Tordjman, M., Liu, Z., Yuce, M., Fauveau, V., Mei, Y., Hadjadj, J., Bolger, I., Almansour, H., Horst, C., Parihar, A. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03726-3 · Cited by 174
A systematic review of immersive virtual reality for industrial skills training
Radhakrishnan, U., Koumaditis, K., & Chinello, F. (2021) · Behaviour & Information Technology · doi:10.1080/0144929x.2021.1954693 · Cited by 200
“A Fleiss Kappa (Fleiss 1971) inter-rater reliability score of 0.578 was calculated on the 802 ratings by three judges, showing moderate agreement (0.41 – 0.60).”
MethodInfluentialArtificial intelligence (AI)-enabled CRM capability in healthcare: The impact on service innovation
Kumar, P., Sharma, S. K., & Dutot, V. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2022.102598 · Cited by 176
Hughes, M. A., & Garrett, D. E. (1990) · Journal of Marketing Research · doi:10.1177/002224379002700206 · Cited by 175
Automatic Recognition of Non-Acted Affective Postures
Kleinsmith, A., Bianchi-Berthouze, N., & Steed, A. (2011) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2010.2103557 · Cited by 174
“Fleiss’ kappa [45] was used to compute the interobserver reliability across the complete set of evaluations as it takes into account multiple raters.”
MethodKappa and Beyond: Is There Agreement?
Dettori, J. R., & Norvell, D. C. (2020) · Global Spine Journal · doi:10.1177/2192568220911648 · Cited by 171
“It was later extended by Fleiss to include multiple observers.(5) For illustration purposes, we will look at the simpler case of 2 observers, acknowledging that the principles are the same for multiple observers.”
MethodYoo, E., Rand, W., Eftekhar, M., & Rabinovich, E. (2016) · Journal of Operations Management · doi:10.1016/j.jom.2016.05.007 · Cited by 173
“We then checked the inter-rater agreement of our and the raters' classi fi cations using Fleiss' kappa (Fleiss, 1971).”
MethodStorr, A., Venetis, C. A., Cooke, S., Kilani, S., & Ledger, W. (2017) · Human Reproduction · doi:10.1093/humrep/dew330 · Cited by 171
“The Fleiss kappa coefficient (k) (Fleiss, 1971) was used in order to evaluate the extent of agreement.”
Skirgård, H., Haynie, H. J., Blasi, D. E., Hammarström, H., Collins, J., Latarche, J. J., Lesage, J., Weber, T., Witzlack-Makarevich, A., Passmore, S., et al. (2023) · Science Advances · doi:10.1126/sciadv.adg6175 · Cited by 169
High Agreement and High Prevalence: The Paradox of Cohen’s Kappa
Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017) · The Open Nursing Journal · doi:10.2174/1874434601711010211 · Cited by 169
PyCM: Multiclass confusion matrix library in Python
Haghighi, S., Jasemi, M., Hessabi, S., & Zolanvari, A. (2018) · Journal of Open Source Software · doi:10.21105/joss.00729 · Cited by 167
“…and a proper tool for post-classification model evaluation that supports most classes and overall statistics parameters (Landis and Koch 1977; Fleiss 1971; Altman 1990; Gwet 2008; Scott 1955; Bennett, Alpert, and Goldstein 1954; Cicchetti 1994; Davies 1980; Kullback and Leibler 1951; Goodman…”
MethodPerceptions, Expectations, and Challenges in Defect Prediction
Wan, Z., Xia, X., Hassan, A. E., Lo, D., Yin, J., & Yang, X. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877678 · Cited by 169
“We then use the Fleiss Kappa measure [19] to examine the agreement between the two labelers.”
MethodA Hierarchically-Labeled Portuguese Hate Speech Dataset
Fortuna, P., Rocha da Silva, J., Soler-Company, J., Wanner, L., & Nunes, S. (2019) · Proceedings of the Third Workshop on Abusive Language Online · doi:10.18653/v1/w19-3510 · Cited by 161
“To check the agreement between the three clas-sifications of every message, we used Fleiss’s Kappa (Fleiss, 1971).”
MethodInterpersonal concomitants and antecedents of depression among college students.
Hokanson, J. E., Rubert, M. P., Welker, R. A., & Hollander, G. R. (1989) · Journal of Abnormal Psychology · doi:10.1037//0021-843x.98.3.209 · Cited by 167
On the stability of feature selection algorithms
NogueiraSarah, SechidisKonstantinos, & BrownGavin (2017) · Journal of Machine Learning Research · doi:10.5555/3122009.3242031 · Cited by 138
Helical CT for the Diagnosis of Extracranial Internal Carotid Artery Dissection
Leclerc, X., Godefroy, O., Salhi, A., Lucas, C., Leys, D., & Pruvo, J. P. (1996) · Stroke · doi:10.1161/01.str.27.3.461 · Cited by 170
The Clinical Differentiation of Bacterial and Fungal Keratitis: A Photographic Survey
Dalmon, C., Porco, T. C., Lietman, T. M., Prajna, N. V., Prajna, L., Das, M. R., Kumar, J. A., Mascarenhas, J., Margolis, T. P., Whitcher, J. P., et al. (2012) · Investigative Opthalmology & Visual Science · doi:10.1167/iovs.11-8478 · Cited by 164
Berth-Jones, J., Grotzinger, K., Rainville, C., Pham, B., Huang, J., Daly, S., Herdman, M., Firth, P., & Hotchkiss, K. (2006) · British Journal of Dermatology · doi:10.1111/j.1365-2133.2006.07389.x · Cited by 165
Cervical vertebrae maturation method morphologic criteria: Poor reproducibility
Nestman, T. S., Marshall, S. D., Qian, F., Holton, N., Franciscus, R. G., & Southard, T. E. (2011) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2011.04.013 · Cited by 164
“However, a more stringent measure of association for use with categorical data is recommended for measuring agreement between judges.(24-27) The reproducibility of the CVM method was recently assessed by Gabriel et al,(1) who attempted to use methods to “eliminate the methodologic shortcomings of previous studies.”
Dressler, W. H., Wilson, D., Clendenning, J., Cramb, R., Keenan, R., Mahanty, S., Bruun, T. B., Mertz, O., & Lasco, R. D. (2017) · Ambio · doi:10.1007/s13280-016-0836-z · Cited by 165
“Tests for heterogeneity includedv2, degrees of freedom,pvalues, and I2, and were reported for all analyses.”
InfluentialChen, D., Matson, M., Macapagal, K., Johnson, E. K., Rosoklija, I., Finlayson, C., Fisher, C. B., & Mustanski, B. (2018) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2017.11.306 · Cited by 165
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
West, P., Bhagavatula, C., Hessel, J., Hwang, J., Jiang, L., Le Bras, R., Lu, X., Welleck, S., & Choi, Y. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.341 · Cited by 156
Statement Validity Analysis and the ‘detection of the truth’
Köhnken, G. (2004) · The Detection of Deception in Forensic Contexts · doi:10.1017/cbo9780511490071.003 · Cited by 165
Interrater reliability of EEG-video monitoring
Benbadis, S. R., LaFrance, W. C., Papandonatos, G. D., Korabathina, K., Lin, K., & Kraemer, H. C. (2009) · Neurology · doi:10.1212/wnl.0b013e3181b78425 · Cited by 165
CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
Zhang, N., Chen, M., Bi, Z., Liang, X., Li, L., Shang, X., Yin, K., Tan, C., Xu, J., Huang, F., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.544 · Cited by 152
Brunt, E. M., Clouston, A. D., Goodman, Z., Guy, C., Kleiner, D. E., Lackner, C., Tiniakos, D. G., Wee, A., Yeh, M., Leow, W. Q., et al. (2022) · Journal of Hepatology · doi:10.1016/j.jhep.2022.01.011 · Cited by 162
The Prevalence and 3-Year Incidence of Dementia in Older Black and White Community Residents
Fillenbaum, G. G., Heyman, A., Huber, M. S., Woodbury, M. A., Leiss, J., Schmader, K. E., Bohannon, A., & Trapp-Moen, B. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00024-9 · Cited by 164
Meyer, G. J., Hilsenroth, M. J., Baxter, D., Exner, J. E., Fowler, J. C., Piers, C. C., & Resnick, J. (2002) · Journal of Personality Assessment · doi:10.1207/s15327752jpa7802_03 · Cited by 164
“& Fleiss, 1987). According to Shrout and Fleiss (1979), a one-way random effects model treats each protocol as if it were scored by a different set of raters who are randomly selected from a larger population of raters.”
MethodInfluentialJagannatha, A., Liu, F., Liu, W., & Yu, H. (2019) · Drug Safety · doi:10.1007/s40264-018-0762-z · Cited by 161
Teufel, S., Siddharthan, A., & Batchelor, C. (2009) · Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 3 - EMNLP '09 · doi:10.3115/1699648.1699696 · Cited by 161
“As agreement measure we choose the Kappa coefficient κ (Fleiss, 1971; Siegel and Castellan, 1988), the agreement measure predominantly used in natural language processing research (Carletta, 1996). κ corrects raw agreement P ( A ) for agreement by chance P ( E ) : No matter how many items or…”
BackgroundMethodOn the pragmatic design of literature studies in software engineering: an experience-based guideline
Kuhrmann, M., Fernández, D. M., & Daneva, M. (2017) · Empirical Software Engineering · doi:10.1007/s10664-016-9492-y · Cited by 161
Implementing a General Framework for Assessing Interrater Agreement in Stata
Klein, D. (2018) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x1801800408 · Cited by 158
“359 for Fleiss’s kappa to 0.390 for Krippendorff’s alpha. All coefficients are statistically significant at the 5% level. Using the Landis and Koch (1977) scale, discussed in section 2.”
InfluentialModeling Agreement among Raters
Tanner, M. A., & Young, M. A. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10477157 · Cited by 161
“See, for example, Cohen (1968), Fleiss et al. (1969), Fleiss (1971), and Fleiss and Cohen (1973).”
BackgroundUsing crowdsourcing for TREC relevance assessment
Alonso, O., & Mizzaro, S. (2012) · Information Processing & Management · doi:10.1016/j.ipm.2012.01.004 · Cited by 160
Aceves-Bueno, E., Adeleye, A. S., Bradley, D., Tyler Brandt, W., Callery, P., Feraud, M., Garner, K. L., Gentry, R., Huang, Y., McCullough, I., et al. (2015) · Ecosystems · doi:10.1007/s10021-015-9842-4 · Cited by 161
“For this reason, the Fleiss’s K coefficient, a conservative index, was also computed, following the procedures described in Fleiss (1971) and Landis and Koch (1977).”
MethodCampbell, R. (2005) · Violence and Victims · doi:10.1891/0886-6708.2005.20.1.55 · Cited by 160
Pearce, M. T., Müllensiefen, D., & Wiggins, G. A. (2010) · Perception · doi:10.1068/p6507 · Cited by 160
Tiling in Interactive Panoramic Video: Approaches and Evaluation
Gaddam, V. R., Riegler, M., Eg, R., Griwodz, C., & Halvorsen, P. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2586304 · Cited by 160
“We then calculated the agreement between participants’ quality ratings for each sequence using Fleiss’s Kappa statistic [56].”
MethodKnowledge-Grounded Dialogue Generation with Pre-trained Language Models
Zhao, X., Wu, W., Xu, C., Tao, C., Zhao, D., & Yan, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.272 · Cited by 152
A systematic review of natural language processing applied to radiology reports.
Casey, A., Davidson, E. M., Poon, M. T. C., Dong, H., Duma, D., Grivas, A., Grover, C., Suárez-Paniagua, V., Tobin, R., Whiteley, W. N., et al. (2021) · UCL Discovery (University College London) · Cited by 160
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160
“Instead, we used Fleiss’s Kappa [32] score.”
Exchange of Stuttering From Function Words to Content Words With Age
Howell, P., Au-Yeung, J., & Sackin, S. (1999) · Journal of Speech, Language, and Hearing Research · doi:10.1044/jslhr.4202.345 · Cited by 159
“The agreement between the two judges on content/function word was 98%, giving a kappa coefficient of .96, which shows a high level of agreement against chance (Fleiss, 1971).”
BackgroundLeys, D., Pruvo, J. P., Parent, M., Vermersch, P., Soetaert, G., Steinling, M., Delacourte, A., Defossez, A., Rapoport, A., & Clarisse, J. (1991) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.54.1.46 · Cited by 159
“The moustache sits down first”: on the acquisition of metonymy
FALKUM, I. L., RECASENS, M., & CLARK, E. V. (2017) · Journal of Child Language · doi:10.1017/s0305000915000720 · Cited by 158
“We used Fleiss’ kappa (Fleiss, 1971), measured using the irr package in R: κ = ·86.”
MethodPacheco, C., García, I., & Reyes, M. (2018) · IET Software · doi:10.1049/iet-sen.2017.0144 · Cited by 158
“Moreover, our validation exercises include the following: • Validation 1 – Inter-rater agreement : According to Fleiss’ Kappa [33], the inter-rater agreement depicts the consistency of the data extraction among the analysed studies when two or more researchers evaluate each study.”
MethodILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation
Malik, V., Sanjay, R., Nigam, S. K., Ghosh, K., Guha, S. K., Bhattacharya, A., & Modi, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.313 · Cited by 292
“We also calculate Fleiss’ kappa (Fleiss, 1971) as 0.”
MethodCharacteristics of Useful Code Reviews: An Empirical Study at Microsoft
Bosu, A., Greiler, M., & Bird, C. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.21 · Cited by 248
“Since we used more than two raters for the transcriptions we calculated interrater reliability using Fleiss’ Kappa [17] on the individually coded responses.”
MethodReliability of Psychiatric Diagnosis: I. A Methodological Review
Helzer, J. E. (1977) · Archives of General Psychiatry · doi:10.1001/archpsyc.1977.01770140019001 · Cited by 158
Hall, J., Kellett, S., Berrios, R., Bains, M. K., & Scott, S. (2016) · The American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2016.06.006 · Cited by 157
Expressing Emotions with the Social Robot Probo
Saldien, J., Goris, K., Vanderborght, B., Vanderfaeillie, J., & Lefeber, D. (2010) · International Journal of Social Robotics · doi:10.1007/s12369-010-0067-6 · Cited by 175
“To measure a certain level of agreement between the users the Fleiss’ Kappa (κ) [13] value was calculated.”
MethodXCOPA: A Multilingual Dataset for Causal Commonsense Reasoning
Ponti, E. M., Glavaš, G., Majewska, O., Liu, Q., Vulić, I., & Korhonen, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.185 · Cited by 152
Campbell, R. (2005) · Violence and Victims · doi:10.1891/vivi.2005.20.1.55 · Cited by 243
Using Adapted Short MASTs for Assessing Parental Alcoholism: Reliability and Validity
Crews, T. M., & Sher, K. J. (1992) · Alcoholism: Clinical and Experimental Research · doi:10.1111/j.1530-0277.1992.tb01420.x · Cited by 157
A systematic literature review of stakeholder identification methods in requirements elicitation
Pacheco, C., & García, I. (2012) · Journal of Systems and Software · doi:10.1016/j.jss.2012.04.075 · Cited by 198
“Our validation exercises included: 1) Inter-rater agreement: According to Fleiss’ Kappa (Fleiss 1971) the interagreement denotes thereby the data extraction consistency between the research studies when two or more researchers assess each paper.”
MethodMeaning and Health: A Systematic Review
Roepke, A. M., Jayawickreme, E., & Riffle, O. M. (2014) · Applied Research in Quality of Life · doi:10.1007/s11482-013-9288-9 · Cited by 156
Hartung, T. J., Friedrich, M., Johansen, C., Wittchen, H., Faller, H., Koch, U., Brähler, E., Härter, M., Keller, M., Schulz, H., et al. (2017) · Cancer · doi:10.1002/cncr.30846 · Cited by 155
“17) 10 40 (32-47) 88 (86-89) 27 (24-31) 15 (11-18) 96 (96-97) 85 (84-86) 0.”
BackgroundPérez, J., Díaz, J., García-Martín, J., & Tabuenca, B. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110657 · Cited by 194
“Finally, it would be interesting to check if, once IC/EC have been refined (k>0,8), the rest of selection process could be conducted by N judges who had not participated in this criteria refinement, as considered by Fleiss's K [15] or Krippendorff's Alpha [22].”
MethodA systematic review of software architecture evolution research
Breivold, H. P., Crnković, I., & Larsson, M. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.06.002 · Cited by 165
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection
Roth, J., Chaudhuri, S., Klejch, O., Marvin, R., Gallagher, A., Kaver, L., Ramaswamy, S., Stopczynski, A., Schmid, C., Xi, Z., et al. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053900 · Cited by 152
“Each track was manually labeled by three an-notators, and the Fleiss’ kappa [16] value over the dataset was 0.72 indicating a high inter-annotator agreement.”
MethodInfluentialAdversarial Training for Weakly Supervised Event Detection
Wang, X., Han, X., Liu, Z., Sun, M., & Li, P. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1105 · Cited by 185
“We use the Fleiss’ kappa (Fleiss, 1971) to measure the annotation consistency among these annotators.”
MethodDreaddit: A Reddit Dataset for Stress Analysis in Social Media
Turcan, E., & McKeown, K. (2019) · Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019) · doi:10.18653/v1/d19-6213 · Cited by 145
“47 , using Fleiss’s Kappa (Fleiss, 1971), considered “moderate agreement” by Landis and Koch (1977).”
MethodChauhan, D. S., R, D. S., Ekbal, A., & Bhattacharyya, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.401 · Cited by 154
“We achieve an overall Fleiss’ (Fleiss, 1971) kappa score of 0.81, which is considered to be reliable.”
MethodInfluentialA Language Modeling Approach for Temporal Information Needs
Berberich, K., Bedathur, S., Alonso, O., & Weikum, G. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-12275-0_5 · Cited by 96
“To measure the degree of agreement between assessors, we computed the Fleiss’ κ statistic [15] of 0.”
MethodBush, Z. M., Longtine, J. A., Cunningham, T. L., Schiff, D., Jane, J. A., Vance, M. L., Thorner, M. O., Laws, E. R., & Lopes, M. B. S. (2010) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2010-0441 · Cited by 153
TRAIT SYNERGISMS AND THE RARITY, EXTIRPATION, AND EXTINCTION RISK OF DESERT FISHES
Olden, J. D., Poff, N. L., & Bestgen, K. R. (2008) · Ecology · doi:10.1890/06-1864.1 · Cited by 153
General Chemistry Students’ Goals for Chemistry Laboratory Coursework
DeKorver, B. K., & Towns, M. H. (2015) · Journal of Chemical Education · doi:10.1021/acs.jchemed.5b00463 · Cited by 170
Twitter sentiment classification for measuring public health concerns
Ji, X., Chun, S. A., Wei, Z., & Geller, J. (2015) · Social Network Analysis and Mining · doi:10.1007/s13278-015-0253-5 · Cited by 164
“We utilized Fleiss’ Kappa (Fleiss 1971) to measure the inter-rater agreement between the three annotators of each tweet.”
MethodContent validity evidences in test development: an applied perspective
Delgado‐Rico, E., Carretero-Dios, H., & Ruch, W. (2012) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-64551 · Cited by 236
“The type of Kappa index used was that applied to categorical Judgments made by multiple Judges (Fleiss, 1971).”
MethodCho, N., Jang, M., Lyou, C. Y., Park, J. S., Choi, H. Y., & Moon, W. K. (2011) · Radiology · doi:10.1148/radiol.11110886 · Cited by 150
Real‐time classification of Twitter trends
Zubiaga, A., Spina, D., Martínez, R., & Fresno, V. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23186 · Cited by 155
“In order to measure the inter-annotator agreement we computed the Fleiss’s kappa coefficient (Fleiss, 1971), obtaining a value of 0.”
MethodInfluentialAI feel you: customer experience assessment via chatbot interviews
Sidaoui, K., Jaakkola, M., & Burton, J. (2020) · Journal of Service Management · doi:10.1108/josm-11-2019-0341 · Cited by 148
Shetty, P., Rajan, A. C., Kuenneth, C., Gupta, S., Panchumarti, L. P., Holm, L., Zhang, C., & Ramprasad, R. (2023) · npj Computational Materials · doi:10.1038/s41524-023-01003-w · Cited by 149
“The Fleiss Kappa metric was computed to be 0.885 and the pairwise Cohen’s Kappa metric to be (0.906, 0.864, 0.887) for each of the three pairs of annotators.”
MethodInfluentialArtificial intelligence for caries and periapical periodontitis detection
Li, S., Liu, J., Zhou, Z., Zhou, Z., Wu, X., Li, Y., Wang, S., Liao, W., Ying, S., & Zhao, Z. (2022) · Journal of Dentistry · doi:10.1016/j.jdent.2022.104107 · Cited by 184
Inter-Coder Agreement in Qualitative Coding: Considerations for its Use
Halpin, S. N. (2024) · American Journal of Qualitative Research · doi:10.29333/ajqr/14887 · Cited by 149
Evaluating neurological outcome measures: the bare essentials.
Hobart, J. C., Lamping, D. L., & Thompson, A. J. (1996) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.60.2.127 · Cited by 149
Detection of intracranial aneurysms with multislice CT: comparison with conventional angiography
Dammert, S., Krings, T., Möller‐Hartmann, W., Ueffing, E., Hans, F., Willmes, K., Mull, M., & Thron, A. K. (2004) · Neuroradiology · doi:10.1007/s00234-003-1155-1 · Cited by 149
“Assessment of interobserver agreement included unweighted kappa values for blinded interpretation of CTA and DSA studies according to Fleiss [ 7 ] and Landis and Koch [8].”
MethodCarotid Artery Anatomy and Geometry as Risk Factors for Carotid Atherosclerotic Disease
Phan, T. G., Beare, R. J., Jolley, D., Das, G., Ren, M., Wong, K., Chong, W., Sinnott, M. D., Hilton, J. E., & Srikanth, V. K. (2012) · Stroke · doi:10.1161/strokeaha.111.645499 · Cited by 148
Dietz, J. S., Chompalov, I., Bozeman, B. L., Lane, E. O., & Park, J. (2000) · Scientometrics · doi:10.1023/a:1010537606969 · Cited by 186
Baxter, D. G., Liu, L., Petrich, S., Gisselman, A. S., Chapple, C., Anders, J. J., & Tumilty, S. (2017) · BMC Cancer · doi:10.1186/s12885-017-3852-x · Cited by 157
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.