Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 201–300 of 1,118
ToTTo: A Controlled Table-To-Text Generation Dataset
Parikh, A. P., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020) · arXiv · doi:10.48550/arxiv.2004.14373 · Cited by 252
“We compute annotator agreement and Fleiss’ kappa (Fleiss, 1971) and (3) final sentence after the secondary grammar correction task.”
MethodDiversity of decision-making models and the measurement of interrater agreement
Uebersax, J. S. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.101.1.140 · Cited by 254
“In fact, a wide range of alternatives have been suggested, including ps, the proportion of specific agreement ( Fleiss, 1971 ) and agreement with majority opinion (Schouten, in press), both of which offer the advantage of providing separate measures of agreement on positive and negative diagnoses, the random error coe…”
BackgroundCoder Reliability and Misclassification in the Human Coding of Party Manifestos
Mikhaylov, S., Laver, M., & Benoit, K. R. (2012) · Political Analysis · doi:10.1093/pan/mpr047 · Cited by 250
“Overall, these results show that regardless of whether coders are compared in the full category tests or on the reduced three-fold “Rile” classification, rater agreement is exceptionally poor by 9In a slightly more lenient set of guidelines, Fleiss, Levin and Paik (2003, 604) following Landis and Koch (1977) proposed…”
To personalize or not to personalize: modeling queries with variation in user intent
Teevan, J., Dumais, S. T., & Liebling, D. J. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390364 · Cited by 250
“As a measure of inter-rater reliability, we calculated Fleiss‟ kappa (κ) [7].”
MethodStaging disease severity in pathologically confirmed cases of frontotemporal dementia
Broe, M., Hodges, J. R., Schofield, E., Shepherd, C. E., Kril, J. J., & Halliday, G. M. (2003) · Neurology · doi:10.1212/01.wnl.0000052685.09194.39 · Cited by 249
Annotating Argument Components and Relations in Persuasive Essays
Stab, C., & Gurevych, I. (2014) · TUbilio (Technical University of Darmstadt) · Cited by 441
A Descriptive Study of the Centralization Phenomenon: A Prospective Analysis
Werneke, M., Hart, D. L., & Cook, D. (1999) · Spine · doi:10.1097/00007632-199904010-00012 · Cited by 247
Three-Point Checklist of Dermoscopy
Soyer, H. P., Argenziano, G., Zalaudek, I., Corona, R., Sera, F., Talamini, R., Barbato, F., Baroni, A., Cicale, L., Di Stefani, A., et al. (2004) · Dermatology · doi:10.1159/000075042 · Cited by 243
Miniati, M., Monti, S., Pratali, L., Di Ricco, G., Marini, C., Formichi, B., Prediletto, R., Michelassi, C., Di Lorenzo, M., Tonelli, L., et al. (2001) · The American Journal of Medicine · doi:10.1016/s0002-9343(01)00693-3 · Cited by 237
Galaxy Zoo: Motivations of Citizen Scientists
Jordan Raddick, M., Bracey, G., Gay, P. L., Lintott, C. J., Cardamone, C., Murray, P., Schawinski, K., Szalay, A. S., & Vandenberg, J. (2013) · Astronomy Education Review · doi:10.3847/aer2011021 · Cited by 217
“This agreement results in a Fleiss Kappa (Fleiss 1971) interrater reliability of 0.661.”
InfluentialShinar, D., Gross, C. R., Price, T. R., Banko, M., Bolduc, P. L., & Robinson, R. G. (1986) · Stroke · doi:10.1161/01.str.17.2.241 · Cited by 242
Data-driven Bayesian network for risk analysis of global maritime accidents
Li, H., Ren, X., & Yang, Z. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2022.108938 · Cited by 237
Sample-size calculations for Cohen's kappa.
Cantor, A. B. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.150 · Cited by 240
“Extensions have also been made to allow for more than two raters (Posner, Sampson, Caplan, Ward, & Chenly, 1990), ordinal data (Fleiss, 1978), and continuous data (Rae, 1988).”
BackgroundA Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple Raters
Berry, K. J., & Mielke, P. W. (1988) · Educational and Psychological Measurement · doi:10.1177/0013164488484007 · Cited by 239
TabFact: A Large-scale Dataset for Table-based Fact Verification
Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., & Wang, W. Y. (2019) · arXiv · doi:10.48550/arxiv.1909.02164 · Cited by 239
“We follow the previous works (Thorne et al., 2018; Bowman et al., 2015) to adopt the Fleiss Kappa (Fleiss, 1971) as an indicator, where Fleiss κ = ¯ p c − ¯ p e 1 − ¯ p e is computed from from the observed agreement ¯ p c and the agreement by chance ¯ p e .”
MethodSummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
“early attempts were unsuccessful, finding that reranking summaries according to an NLI model can lead to an increase in consistency errors (Falke et al., 2019), or that out-of-the-box NLI models obtain 52% accuracy at the binary classification task of inconsistency detection, only slightly above random guessing (Krysc…”
BackgroundMethodResultInfluentialthe Canada PRIME Plus Team, Patey, A. M., Islam, R., Francis, J. J., Bryson, G. L., & Grimshaw, J. M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-52 · Cited by 237
“Subsequent coding of the remaining interviews was completed independently and Fleiss’s Kappa (κ) was calculated for all domains and interviews to assess whether the two researchers coded the same response into the same domain [26,27].”
MethodRegularized estimation of large-scale gene association networks using graphical Gaussian models
Krämer, N., Schäfer, J., & Boulesteix, A.-L. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-384 · Cited by 237
“These frequencies are summarized using Fleiss’ -score [49] which measures the degree of agreement among the R subsamples of the data.”
Extension of the Kappa Coefficient
Kraemer, H. C. (1980) · Biometrics · doi:10.2307/2529972 · Cited by 237
Findings of the 2013 Workshop on Statistical Machine Translation
Bojar, O., Buck, C., Callison-Burch, C., Federmann, C., Haddow, B., Koehn, P., Monz, C., Post, M., Soricut, R., & Specia, L. (2013) · UvA-DARE (University of Amsterdam) · Cited by 236
Smith, T. E., Sheridan, S. M., Kim, E. M., Park, S., & Beretvas, S. N. (2020) · Educational Psychology Review · doi:10.1007/s10648-019-09509-w · Cited by 231
The Female Sexual Function Index: Translation and Validation of an Iranian Version
Fakhri, A., Pakpour, A. H., Burri, A., Morshedi, H., & Zeidi, I. M. (2012) · The Journal of Sexual Medicine · doi:10.1111/j.1743-6109.2011.02553.x · Cited by 235
Fleiss’ kappa statistic without paradoxes
Falotico, R., & Quatto, P. (2015) · Quality & Quantity · doi:10.1007/s11135-014-0003-1 · Cited by 233
Variable selection for generalized canonical correlation analysis
Tenenhaus, A., Philippe, C., Guillemot, V., Le Cao, K.-A., Grill, J., & Frouin, V. (2014) · Biostatistics · doi:10.1093/biostatistics/kxu001 · Cited by 234
“To evaluate the stability of the signatures, we propose using the Downloaded from https://academic.oup.com/biostatistics/article-abstract/15/3/569/224062 by guest on 29 July 2018 Fleiss’ κ indicator defined in Fleiss (1971).”
MethodWhat developers want and need from program analysis: an empirical study
Christakis, M., & Bird, C. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970347 · Cited by 219
“To alleviate this threat, a random sample of 20% of these incidents were coded by another researcher independently and inter-rater reliability [36] was calculated using Fleiss κ on the results [33].”
MethodGotwals, J. K., Stoeber, J., Dunn, J. G. H., & Stoll, O. (2012) · Canadian Psychology / Psychologie canadienne · doi:10.1037/a0030288 · Cited by 228
“A substantial level of agreement among the four judges was obtained as indicated by a Fleiss kappa of .75 (Fleiss, 1971; Landis & Koch, 1977).”
Observer variation in the assessment of dysplasia in ulcerative colitis
DIXON, M. F., BROWN, L. J. R., GILMOUR, H. M., PRICE, A. B., SMEETON, N. C., TALBOT, I. C., & WILLIAMS, G. T. (1988) · Histopathology · doi:10.1111/j.1365-2559.1988.tb02055.x · Cited by 230
Measuring interrater reliability among multiple raters: An example of methods for nominal data
Posner, K. L., Sampson, P. D., Caplan, R. A., Ward, R. J., & Cheney, F. W. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090917 · Cited by 229
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
Guan, J., Huang, F., Zhao, Z., Zhu, X., & Huang, M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00302 · Cited by 228
Computer Recognition of Regional Lung Disease Patterns
UPPALURI, R., HOFFMAN, E. A., SONKA, M., HARTLEY, P. G., HUNNINGHAKE, G. W., & McLENNAN, G. (1999) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.160.2.9804094 · Cited by 227
“For the third reading, the kappa statistic ( 25 ) of agreement between the observers was also reported.”
BackgroundMeasuring agreement in medical informatics reliability studies
Hripcsak, G., & Heitjan, D. F. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(02)00500-2 · Cited by 226
“It is also possible to define kappa when different groups of raters assess each case [17], even when the number of raters varies per case [10].”
BackgroundAn Examination of the Use of Large Language Models to Aid Analysis of Textual Data
(2024) · doi:10.1177/16094069241231168 · Cited by 227
“Another example of an inter-rater statistic is Fleiss ’ kappa, which is used when there are more than two coders (Fleiss, 1971).”
Inequalities between multi-rater kappas
Warrens, M. J. (2010) · Advances in Data Analysis and Classification · doi:10.1007/s11634-010-0073-4 · Cited by 226
“(nominal) categories, such as demographic groups, psychodiagnostic classifications or production faults (Fleiss 1971; Zwick 1988; De Mast 2007).”
InfluentialA revised teacher rating scale for Reactive and Proactive Aggression
Brown, K., Atkins, M. S., Osborne, M. L., & Milnamow, M. (1996) · Journal of Abnormal Child Psychology · doi:10.1007/bf01441569 · Cited by 224
“The kappa coefficient (Fleiss, 1971) for the total antisocial scale was .”
BackgroundNot all systematic reviews are created equal: Considerations for appraisal
Schlosser, R. W., Wendt, O., & Sigafoos, J. (2007) · Evidence-Based Communication Assessment and Intervention · doi:10.1080/17489530701560831 · Cited by 223
“Fleiss, J. L. (1971). Measuring nominal scale agreement among”
BackgroundMethodYen, K., Riegert, A., & Gorelick, M. H. (2008) · Pediatric Emergency Care · doi:10.1097/pec.0b013e3181666f32 · Cited by 221
“For the 10% (65 subjects) of attempts where 2 nurses completed the subjective portion of the data forms, interobserver agreement was measured with a weighted 0 statistic.(4) We next developed a multivariate predictive model for difficult peripheral intravenous placement.”
MethodAutomated Skeletal Classification with Lateral Cephalometry Based on Artificial Intelligence
Yu, H. J., Cho, S. R., Kim, M. J., Kim, W. H., Kim, J. W., & Choi, J. (2020) · Journal of Dental Research · doi:10.1177/0022034520901715 · Cited by 221
A dataset of neonatal EEG recordings with seizure annotations
Stevenson, N. J., Tapani, K., Lauronen, L., & Vanhatalo, S. (2019) · Scientific Data · doi:10.1038/sdata.2019.39 · Cited by 219
Kawesha, A., Ghaneh, P., Andrén‐Sandberg, Å., Ögraed, D., Skar, R., Dawiskiba, S., Evans, J. D., Campbell, F., Lemoine, N. R., & Neoptolemos, J. P. (2000) · International Journal of Cancer · doi:10.1002/1097-0215(20001120)89:6<506::aid-ijc7>3.0.co;2-f · Cited by 152
Gottesman, I. I., & Shields, J. (1976) · Schizophrenia Bulletin · doi:10.1093/schbul/2.3.360 · Cited by 221
“Kety (1974b) does report a measure of reliability for the concept of definite plus uncertain schizophrenia for three judges; the value of kappa (Fleiss 1971) is .”
BackgroundSorror, M. L., Logan, B. R., Zhu, X., Rizzo, J. D., Cooke, K. R., McCarthy, P. L., Ho, V. T., Horowitz, M. M., & Pasquini, M. C. (2015) · Biology of Blood and Marrow Transplantation · doi:10.1016/j.bbmt.2015.04.004 · Cited by 217
Gallagher, R. M., Kirkham, J. J., Mason, J. R., Bird, K. A., Williamson, P. R., Nunn, A. J., Turner, M. A., Smyth, R. L., & Pirmohamed, M. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028096 · Cited by 218
“To supplement the pair-wise kappas, a global kappa score measuring nominal scale agreement across multiple assessors was calculated with 95% confidence intervals [19].”
MethodHow I assess comorbidities before hematopoietic cell transplantation
Sorror, M. L. (2013) · Blood · doi:10.1182/blood-2012-09-455063 · Cited by 217
Inter-Rater Reliability Methods in Qualitative Case Study Research
Cole, R. (2024) · Sociological Methods & Research · doi:10.1177/00491241231156971 · Cited by 213
“Similarly, Fleiss ’ kappa (Fleiss 1971) can be used with nominal data and was applied by Su et al. (2014).”
Inkpen, D. Z., & Association for Computing Machinery-Digital Library (2010) · Cited by 220
Interobserver Agreement in Behavioral Research: Importance and Calculation
Watkins, M. W., & Pacheco, M. (2000) · Journal of Behavioral Education · doi:10.1023/a:1012295615144 · Cited by 213
“Kappa (k) was later expanded by Fleiss ( 1971) to allow generalization to situations using multiple observers who need not necessarily be the same throughout the study.”
MethodInfluentialAn international study to increase concordance in Ki67 scoring
Polley, M.-Y. C., Leung, S. C. Y., Gao, D., Mastropasqua, M. G., Zabaglo, L. A., Bartlett, J. M. S., McShane, L. M., Enos, R. A., Badve, S. S., Bane, A. L., et al. (2015) · Modern Pathology · doi:10.1038/modpathol.2015.38 · Cited by 214
Subthreshold depression in children and adolescents – a systematic review
Wesselhoeft, R., Sørensen, M. J., Heiervang, E. R., & Bilenberg, N. (2013) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.06.010 · Cited by 214
CONSISTENCY OF AO FRACTURE CLASSIFICATION FOR THE DISTAL RADIUS
Kreder, H. J., Hanel, D. P., McKee, M., Jupiter, J., McGillivary, G., & Swiontkowski, M. F. (1996) · The Journal of Bone and Joint Surgery. British volume · doi:10.1302/0301-620x.78b5.0780726 · Cited by 214
“Since this original description, additional statistics have been described with more flexibility (Fleiss 1971; Fleiss, Nee and Landis 1979; O’Connell and Dobson 1984; Posner et al 1990).”
MethodDetecting and Monitoring Hate Speech in Twitter
Pereira-Kohatsu, J. C., Quijano-Sánchez, L., Liberatore, F., & Camacho-Collados, M. (2019) · Sensors · doi:10.3390/s19214654 · Cited by 214
“The two main statistical measures of the agreement percentage, popular in labeling processes and areas such as psychology and education [76], are Cohens’ kappa [77], which measures the degree of agreement between two raters, and Fliess’ kappa [78], which makes it possible to measure the concordance among two or more r…”
Background(2020) · Progress in Molecular Biology and Translational Science · doi:10.1016/bs.pmbts.2020.04.003 · Cited by 212
A systematic review of natural language processing applied to radiology reports
Casey, A., Davidson, E., Poon, M., Dong, H., Duma, D., Grivas, A., Grover, C., Suárez-Paniagua, V., Tobin, R., Whiteley, W., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01533-7 · Cited by 206
“Fleiss’ kappa [17] agreement between re-96 viewers was 0.70, indicating substantial agreement [18].”
InfluentialA Retrospective Analysis of the Fake News Challenge Stance Detection Task
Hanselowski, A., Pvs, A., Schiller, B., Caspelherr, F., Chaudhuri, D., Meyer, C. M., & Gurevych, I. (2018) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.1806.05180 · Cited by 333
“686 (Fleiss, 1971), which is substantial and allows drawing tentative conclusions (Artstein and Poesio, 2008).”
BackgroundEisenbeiss, S. A., van Knippenberg, D., & Fahrbach, C. M. (2015) · Journal of Business Ethics · doi:10.1007/s10551-014-2124-9 · Cited by 208
“The Kappa coefficient for organizational ethics programs was 0.64 and clearly exceeded the critical aggregation value of 0.40 (Fleiss 1971).”
MethodResultInfluentialDFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild
Jiang, X., Zong, Y., Zheng, W., Tang, C., Xia, W., Lu, C., & Liu, J. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413620 · Cited by 207
“In this section, we discuss the quality of emotion annotation based on Fleiss’s Kappa test [10].”
MethodBerg, W. A., D’Orsi, C. J., Jackson, V. P., Bassett, L. W., Beam, C. A., Lewis, R. S., & Crewson, P. E. (2002) · Radiology · doi:10.1148/radiol.2243011626 · Cited by 208
Signal-based and expectation-based factors in the perception of prosodic prominence
Cole, J., Mo, Y., & Hasegawa-Johnson, M. (2010) · Laboratory Phonology · doi:10.1515/labphon.2010.022 · Cited by 208
Jousse-Joulin, S., D'Agostino, M. A., Nicolas, C., Naredo, E., Ohrndorf, S., Backhaus, M., Tamborrini, G., Chary-Valckenaere, I., Terslev, L., Iagnocco, A., et al. (2019) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2019-215024 · Cited by 204
Do developers feel emotions? an exploratory analysis of emotions in software artifacts
Murgia, A., Tourani, P., Adams, B., & Ortu, M. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597086 · Cited by 200
“To measure the degree of inter-rater agreement on identified emotions, we calculate either Cohen’s κ value [10] (two raters) or Fleiss’ κ value [14] (more than two raters).”
MethodDefining language networks from resting‐state fMRI for surgical planning—a feasibility study
Tie, Y., Rigolo, L., Norton, I. H., Huang, R. Y., Wu, W., Orringer, D., Mukundan, S., & Golby, A. J. (2014) · Human Brain Mapping · doi:10.1002/hbm.22231 · Cited by 207
“…group ICA output, the free-marginal multi-rater kappa coefficient ( κ free ) was calculated [Brennan and Prediger, 1981; Randolph, 2005]. κ free is an alternative to Fleiss’ multirater kappa [Fleiss, 1971] and was used because the number of components assigned into each category was not restricted.”
MethodEvaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207
“Fleiss’ κ (Fleiss, 1971) can measure agreement between multiple evaluators.”
BackgroundOntoLearn Reloaded: A Graph-Based Algorithm for Taxonomy Induction
Velardi, P., Faralli, S., & Navigli, R. (2013) · Computational Linguistics · doi:10.1162/coli_a_00146 · Cited by 206
“Brank, Mladenic, and Grobelnik (2006) exploit the analogy between ontology learning and unsupervised clustering, and propose OntoRand, a modified version of the Rand Index (Rand 1971) for computing the similarity between ontologies.”
A method to quantify major themes of scientific literacy in science textbooks
Chiappetta, E. L., Fillman, D. A., & Sethna, G. H. (1991) · Journal of Research in Science Teaching · doi:10.1002/tea.3660280808 · Cited by 205
A reference collection for web spam
Castillo, C., Donato, D., Becchetti, L., Boldi, P., Leonardi, S., Santini, M., & Vigna, S. (2006) · ACM SIGIR Forum · doi:10.1145/1189702.1189703 · Cited by 202
Drug repurposing for COVID-19 via knowledge graph completion
Zhang, R., Hristovski, D., Schutte, D., Kastrin, A., Fiszman, M., & Kilicoglu, H. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103696 · Cited by 204
“Fleiss ’ κ was used in this case, as more than two annotators were involved in annotation [89].”
Statistical methods for assessing agreement for ordinal data
Jakobsson, U., & Westergren, A. (2005) · Scandinavian Journal of Caring Sciences · doi:10.1111/j.1471-6712.2005.00368.x · Cited by 203
Information needs in bug reports: improving cooperation between developers and users
Breu, S., Premraj, R., Sillito, J., & Zimmermann, T. (2010) · Proceedings of the 2010 ACM conference on Computer supported cooperative work · doi:10.1145/1718918.1718973 · Cited by 204
“We measured inter-rater agreement for the first author’s categorization on a simple random sample of 100 cards with a closed card sort and two additional raters (third and fourth author); the Fleiss’ Kappa [9] value among the three raters was 0.”
MethodAutomatic detection of bad smells in code: An experimental assessment.
Arcelli Fontana, F., Braione, P., & Zanoni, M. (2012) · The Journal of Object Technology · doi:10.5381/jot.2012.11.2.a5 · Cited by 203
Statistical methods for assessing agreement
Fitzmaurice, G. (2002) · Nutrition · doi:10.1016/s0899-9007(02)00802-x · Cited by 202
Affective Neural Response Generation
Asghar, N., Poupart, P., Hoey, J., Jiang, X., & Mou, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-76941-7_12 · Cited by 196
“For words missing in this dictionary, such as stop words and proper nouns, we set the VAD vector to be the neutral vector η = [5, 1, 5], because these words are neutral in pleasantness (V) and power (D), and evoke no arousal (A).”
BackgroundMethodInfluentialOtt, G., Ziepert, M., Klapper, W., Horn, H., Szczepanowski, M., Bernd, H.-W., Thorns, C., Feller, A. C., Lenze, D., Hummel, M., et al. (2010) · Blood · doi:10.1182/blood-2010-03-276766 · Cited by 202
API method recommendation without worrying about the task-API knowledge gap
Huang, Q., Xia, X., Xing, Z., Lo, D., & Wang, X. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238191 · Cited by 192
“We use Fleiss Kappa [18] to measure the agreement between the two labelers.”
Confocal Scanning Laser Microscopy Evidence of Biofilms in Patients With Chronic Rhinosinusitis
Psaltis, A. J., Ha, K. R., Beule, A. G., Tan, L. W., & Wormald, P. (2007) · The Laryngoscope · doi:10.1097/mlg.0b013e31806009b0 · Cited by 200
Artstein, R. (2017) · Handbook of Linguistic Annotation · doi:10.1007/978-94-024-0881-2_11 · Cited by 190
“…proceed with the coefficients that are the most appropriate for gauging the reliability of the annotation process, that is Fleiss’s κ and Krippendorff’s α. Note: The term “kappa” (κ) may refer to several distinct agreement coefficients, most commonly those of Cohen (1960) and Fleiss (1971).”
BackgroundJournal of Counselor Preparation and Supervision
(2024) · Journal of Counselor Preparation and Supervision · doi:10.7729/00.5277 · Cited by 200
White-spot lesions during multibracket appliance treatment: A challenge for clinical excellence
Enaia, M., Bock, N., & Ruf, S. (2011) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2010.12.016 · Cited by 198
Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity
Gwet, K. L. (2002) · Cited by 195
Lyneham, H. J., & Rapee, R. M. (2005) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-200503000-00012 · Cited by 198
“Agreement between the gold standard, parent face-to-face- and telephone-derived diagnoses were determined using thek statistic formultiple raters (Fleiss, 1971).”
MethodUnderstanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.383 · Cited by 197
Towards a Consensus-Based Classification of Childhood Arterial Ischemic Stroke
Bernard, T. J., Manco-Johnson, M. J., Lo, W., MacKay, M. T., Ganesan, V., deVeber, G., Goldenberg, N. A., Armstrong-Wells, J., Dowling, M. M., Roach, E. S., et al. (2012) · Stroke · doi:10.1161/strokeaha.111.624585 · Cited by 195
“28 In addition 95% confidence intervals were determined using the Wald method.”
MethodThe compendium of self-enactable techniques to change and self-manage motivation and behaviour v.1.0
Knittle, K., Heino, M., Marques, M. M., Stenius, M., Beattie, M., Ehbrecht, F., Hagger, M. S., Hardeman, W., & Hankonen, N. (2020) · Nature Human Behaviour · doi:10.1038/s41562-019-0798-9 · Cited by 195
Piqueras, J. A., Martín-Vivar, M., Sandin, B., San Luis, C., & Pineda, D. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.04.022 · Cited by 194
Learning to Select Knowledge for Response Generation in Dialog Systems
Lian, R., Xie, M., Wang, F., Peng, J., & Wu, H. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/706 · Cited by 188
Measuring Genetic Differentiation from Pool-seq Data
Hivert, V., Leblois, R., Petit, E. J., Gautier, M., & Vitalis, R. (2018) · Genetics · doi:10.1534/genetics.118.300900 · Cited by 191
The Reliability of Dichotomous Judgments: Unequal Numbers of Judges per Subject
Fleiss, J. L., & Cuzick, J. (1979) · Applied Psychological Measurement · doi:10.1177/014662167900300410 · Cited by 191
Self-citationReactions to racial discrimination: Emotional stress and help-seeking behaviors.
Carter, R. T., & Forsyth, J. (2010) · Psychological Trauma: Theory, Research, Practice, and Policy · doi:10.1037/a0020102 · Cited by 190
Summarizing software artifacts: a case study of bug reports
Rastkar, S., Murphy, G. C., & Murray, G. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806872 · Cited by 190
“We use the kappa test to measure the level of agreement between the annotators [9].”
MethodThe Evolution of Sustainability Measurement Research
Mura, M., Longo, M., Micheli, P., & Bolzani, D. (2018) · International Journal of Management Reviews · doi:10.1111/ijmr.12179 · Cited by 190
“47) (Fleiss 1971), and disagreements resolved through discussion – this process allowed us to determine further selection criteria as detailed in Table 2; (3) each author individually reviewed around 4960 articles, considering titles and abstracts; (4) the documents selected were reciprocally evaluated (i.”
InfluentialKnowledge discovery in metabolomics: An overview of MS data handling
Boccard, J., Veuthey, J., & Rudaz, S. (2010) · Journal of Separation Science · doi:10.1002/jssc.200900609 · Cited by 189
Beliefs and biases in web search
White, R. (2013) · Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/2484028.2484053 · Cited by 183
“We found a high level of agreement between the physicians (both κ ≥ 0.886), as well as between the two physicians and the consensus labels provided by the crowd-sourced judges (both Fleiss’ multi-rater κ ≥ 0.853 [13]).”
ResultInfluentialIntegration and Sealing Over: Clinically Distinct Recovery Styles From Schizophrenia
McGlashan, T. H. (1975) · Archives of General Psychiatry · doi:10.1001/archpsyc.1975.01760280067006 · Cited by 189
Ring, E. A., Dare, E. A., Crotty, E. A., & Roehrig, G. H. (2017) · Journal of Science Teacher Education · doi:10.1080/1046560x.2017.1356671 · Cited by 188
“Because there were multiple coders and the nature of the data was nominal, interrater reliability was calculated using Fleiss’s kappa (Fleiss, 1971) and determined to be sufficient (κ = .593).”
MethodReformulating Unsupervised Style Transfer as Paraphrase Generation
Krishna, K., Wieting, J., & Iyyer, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.55 · Cited by 185
Diagnosing sepsis is subjective and highly variable: A survey of intensivists using case vignettes.
Rhee, C., Kadri, S. S., Danner, R. L., Suffredini, A. F., Massaro, A. F., Kitch, B. T., Lee, G., & Klompas, M. (2016) · Critical Care · doi:10.1186/s13054-016-1266-9 · Cited by 187
“for categorical ratings [10].”
Temporal lobe rating scale: application to Alzheimer's disease and frontotemporal dementia
Galton, C. J. (2001) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.70.2.165 · Cited by 188
Criteria For Evaluating Usability Evaluation Methods
Hartson, H. R., Andre, T. S., & Williges, R. C. (2003) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1501_13 · Cited by 186
“Several authors, such as Gray and Salzman (1998), have alluded to the concepts of hits, misses, false alarms, and correct rejections in the context of UEM outputs, concepts closely related to thoroughness, validity, and effectiveness.”
BackgroundWhich instruments to support diagnosis of depression have sufficient accuracy? A systematic review
Pettersson, A., Boström, K. B., Gustavsson, P., & Ekselius, L. (2015) · Nordic Journal of Psychiatry · doi:10.3109/08039488.2015.1008568 · Cited by 187
Knowledge matters: The effect of tactical descriptions on negotiation behavior and outcome.
Weingart, L. R., Hyder, E. B., & Prietula, M. J. (1996) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.70.6.1205 · Cited by 185
Overcoming Stickiness: How the Timing of Knowledge Transfer Methods Affects Transfer Difficulty
Szulanski, G., Ringov, D., & Jensen, R. J. (2016) · Organization Science · doi:10.1287/orsc.2016.1049 · Cited by 185
Heikkila, T., Pierce, J. J., Gallaher, S., Kagan, J., Crow, D. A., & Weible, C. M. (2014) · Review of Policy Research · doi:10.1111/ropr.12058 · Cited by 184
“10 According to Fleiss (1971), a Cohen’s Kappa measurement of agreement at or above 0.”
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.