Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
“Koch [1975a, 1975b], numerous measures of observer agreement have been proposed for categorical data, e.g., Goodman and Kruskal [1954], Cohen [1960, 1968], Fleiss [1971], Light [1971], and Cicchetti [1972].”
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847
“93 (Fleiss’ kappa is a statistical measure of the reliability agreement between annotators [52]).”
BackgroundIs the CVI an acceptable indicator of content validity? Appraisal and recommendations
Polit, D. F., Beck, C. T., & Owen, S. V. (2007) · Research in Nursing & Health · doi:10.1002/nur.20199 · Cited by 5,273
“Wynd et al. did not provide actual examples of their computations of multi-rater kappa (which is an extension of Cohen’s kappa developed by Fleiss, 1971), but they indicated that they used the basic formula for kappa, which is: k ¼ ProportionAgreement ProportionChance agreement 1 ProportionChance…”
MethodThe Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225
“Fleiss (1971) generalized Scott’s π to many observers, calling it kappa.”
MethodThe kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884
“For fully-crossed designs with three or more coders, Light (1971) suggests computing kappa for all coder pairs then using the arithmetic mean of these estimates to provide an overall index of agreement.”
MethodInfluentialContent Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
“The index has been adapted for multiple coders and cases in which different coders evaluate different units (Fleiss, 1971).”
BackgroundCROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON
Mohammad, S. M., & Turney, P. D. (2013) · Computational Intelligence · doi:10.1111/j.1467-8640.2012.00460.x · Cited by 2,634
“However, it is appropriate only when the same judges annotate all the instances (Fleiss 1971).”
BackgroundInfluentialComputing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
“…X n i ¼ 1 X q k ¼ 1 r ik ð r ik 2 1 Þ r ð r 2 1 Þ ; and p e j p ¼ X q k ¼ 1 ^p 2 k ; with ^p k ¼ 1 n X n i ¼ 1 r ik r : ð 31Þ Concerning the estimation of the variance of ^g p ; Fleiss (1971) suggested the following variance estimatoru nder the hypothesis of no agreement between ratersb eyond…”
MethodLivingstone, S. R., & Russo, F. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0196391 · Cited by 1,883
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831
“raters [99].”
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture Notes in Computer Science · doi:10.1007/11736790_9 · Cited by 1,628
“After the reconciliation phase, the inter-annotator agreement calculated using the Kappa statistics (Siegel and Castellan, 1988; Fleiss, 1971) was 98.”
MethodInter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Reliability of Nominal Data Based on Qualitative Judgments
Perreault, W. D., & Leigh, L. E. (1989) · Journal of Marketing Research · doi:10.2307/3172601 · Cited by 1,575
Detecting and monitoring depression with a two-item questionnaire (PHQ-2)
Löwe, B., Kroenke, K., & Gräfe, K. (2005) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2004.09.006 · Cited by 1,473
“Interrater reliability for the SCID, as assessed by reviewing videotapes of 60 original interviews [32], was j=0.”
MethodDeCuir-Gunby, J. T., Marshall, P. L., & McCulloch, A. W. (2011) · Field Methods · doi:10.1177/1525822x10388468 · Cited by 1,456
“The third and most popular approach to calculating reliability is to use Cohen’s kappa coefficient (Cohen 1960) or Fleiss’s kappa statistic (Fleiss 1971) for use with nominal variables.”
MethodShoemaker, S. J., Wolf, M. S., & Brach, C. (2014) · Patient Education and Counseling · doi:10.1016/j.pec.2014.05.027 · Cited by 1,322
“External consistency—Inter-rater reliability (IRR) was used to assess the external consistency of the PEMAT, using percentage agreement and either Fleiss’ kappa for more than 2 raters [26] or Cohen’s kappa for 2 raters [27].”
MethodFEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343
“We calculated the Fleiss κ score (Fleiss, 1971) to be 0.6841 which we consider encouraging given the complexity of the task.”
BackgroundMethodInfluentialMorphometric analysis of white matter lesions in MR images: method and validation
Zijdenbos, A. P., Dawant, B. M., Margolin, R. A., & Palmer, A. C. (1994) · IEEE Transactions on Medical Imaging · doi:10.1109/42.363096 · Cited by 1,265
YAGO2: A spatially and temporally enhanced knowledge base from Wikipedia
Hoffart, J., Suchanek, F. M., Berberich, K., & Weikum, G. (2013) · Artificial Intelligence · doi:10.1016/j.artint.2012.06.001 · Cited by 1,232
“Using the test procedure and the variance estimator described by Fleiss [21], we find that we can reject the null hypothesis that there is no agreement among annotators beyond chance, at any significance level larger than 7%.”
MethodDermoscopy of pigmented skin lesions: Results of a consensus meeting via the Internet
Argenziano, G., Soyer, H. P., Chimenti, S., Talamini, R., Corona, R., Sera, F., Binder, M., Cerroni, L., De Rosa, G., Ferrara, G., et al. (2003) · Journal of the American Academy of Dermatology · doi:10.1067/mjd.2003.281 · Cited by 1,208
LOWE, B. (2004) · Journal of Affective Disorders · doi:10.1016/s0165-0327(02)00237-9 · Cited by 1,060
“Inter-rater reliability, measured as coefficient j (95% CI) (Fleiss, 1971), for the SCID diagnosis of ‘major depressive disorder’ was 0.88 (0.47–1.0).”
MethodInfluentialGuo, Y., Barnes, S. J., & Jia, Q. (2017) · Tourism Management · doi:10.1016/j.tourman.2016.09.009 · Cited by 1,039
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1547 · Cited by 1,028
“As this requires a multi-annotator metric over a multi-label task, we used Fleiss’ kappa metric (Fleiss, 1971) per single dialogue act.”
MethodDeep learning algorithms for detection of critical findings in head CT scans: a retrospective study
Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N. G., Venugopal, V. K., Mahajan, V., Rao, P., & Warier, P. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31645-3 · Cited by 1,020
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1152 · Cited by 978
Interrater Reliability of the NIH Stroke Scale
Goldstein, L. B., Bertels, C., & Davis, J. N. (1989) · Archives of Neurology · doi:10.1001/archneur.1989.00520420080026 · Cited by 986
Beyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976
“The application of a variance formula such as that given by Fleiss et al. (1969) may lead to unrealistically narrow confidence intervals for kappa in this context, and spuriously high rejection rates for tests against zero.”
BackgroundMethodInfluentialBest Practices in Quantitative Methods
Osborne, J. (2008) · Sage eBooks · doi:10.4135/9781412995627 · Cited by 942
Liu, Y., Ho, R. C.-M., & Mak, A. (2012) · Journal of Affective Disorders · doi:10.1016/j.jad.2011.08.003 · Cited by 948
“) in terms of study inclusion or exclusion and quality of papers were assessed based on the Fleiss kappa (k)-statistics (Fleiss, 1971).”
MethodHerjanic, B., & Reich, W. (1982) · Journal of Abnormal Child Psychology · doi:10.1007/bf00912324 · Cited by 940
“The kappa statistic (see Cohen, 1960; Fleiss, 1971 ) varies from negative values for less than chance agreement, through 0 for chance agreement, to 1.0 for perfect agreement Depending upon what is being compared, each investigator determines what is an acceptable level of kappa.”
BackgroundInfluentialA New Paradigm for Systematic Literature Reviews in Supply Chain Management
Durach, C. F., Kembro, J., & Wieland, A. (2017) · Journal of Supply Chain Management · doi:10.1111/jscm.12145 · Cited by 924
Navis, C., & Glynn, M. A. (2010) · Administrative Science Quarterly · doi:10.2189/asqu.2010.55.3.439 · Cited by 923
“Because more than two raters were used, we used Fleiss’s kappa (Fleiss, 1971); we calculated k = 0.97, signifying almost perfect agreement.”
MethodInfluentialThe Handbook of Research Synthesis
Ziegel, E. R., Cooper, H., & Hedges, L. (1995) · Technometrics · doi:10.2307/1269651 · Cited by 927
Interrater agreement and interrater reliability: Key concepts, approaches, and applications
Gisev, N., Bell, J. S., & Chen, T. F. (2013) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2012.04.004 · Cited by 924
A Factor Analytic Study of the Boredom Proneness Scale
Vodanovich, S. J., & Kass, S. J. (1990) · Journal of Personality Assessment · doi:10.1207/s15327752jpa5501 · Cited by 871
Interrater reliability and agreement of subjective judgments.
Tinsley, H. E., & Weiss, D. J. (1975) · Journal of Counseling Psychology · doi:10.1037/h0076640 · Cited by 852
“Fleiss (1971) provides a computational example.”
BackgroundMethodInfluentialMeta-analysis of action video game impact on perceptual, attentional, and cognitive skills.
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018) · Psychological Bulletin · doi:10.1037/bul0000130 · Cited by 807
“The agreement between the three raters involved in the coding of this moderator was good (Cohen’s (cid:2) .68; Fleiss (cid:4) (cid:2) .70 and Krippendorf (cid:5) (cid:2) .70; Cohen, 1960; Davies & Fleiss, 1982; Fleiss, 1971; Krippendorf, 1980).”
BackgroundReliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x04266540 · Cited by 829
“Extensions of Cohen’s kappa exist that allows for multiple coders, and further research should examine how the number of coders may effect the coding process ( Fleiss 1971; Fleiss, Nee, and Landis 1979; Conger 1980; Fleiss 1981:225–34).”
BackgroundSocial Media, Knowledge Sharing, and Innovation: Toward a Theory of Communication Visibility
Leonardi, P. M. (2014) · Information Systems Research · doi:10.1287/isre.2014.0536 · Cited by 815
Levy, K. N., Meehan, K. B., Kelly, K. M., Reynoso, J. S., Weber, M., Clarkin, J. F., & Kernberg, O. F. (2006) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.74.6.1027 · Cited by 817
“All kappa and ICC coefficients were in the good to excellent range (Fleiss, 1971).”
BackgroundScreening psychischer Störungen mit dem “Gesundheitsfragebogen für Patienten (PHQ-D)“
Gräfe, K., Zipfel, S., Herzog, W., & Löwe, B. (2004) · Diagnostica · doi:10.1026/0012-1924.50.4.171 · Cited by 801
Two Quantitative Approaches for Estimating Content Validity
Wynd, C. A., Schmidt, B., & Schaefer, M. A. (2003) · Western Journal of Nursing Research · doi:10.1177/0193945903252998 · Cited by 794
“An overall measure of agreement is provided based on an average of the pairwise agreements (Antonakos & Colling, 2001; Bishop, Feinberg, & Holland, 1975; Fleiss, 1971; Siegel & Castellan, 1988).”
BackgroundMethodUnified Parkinson's disease rating scale characteristics and structure
Martínez‐Martín, P., Gil‐Nagel, A., Gracia, L. M., Gómez, J. B., Martínez‐Sarriés, J., Bermejo, F., & The Cooperative Multicentric Group (1994) · Movement Disorders · doi:10.1002/mds.870090112 · Cited by 796
Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
Zhou, H., Huang, M., Zhang, T., Zhu, X., & Liu, B. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11325 · Cited by 763
Tirunillai, S., & Tellis, G. J. (2014) · Journal of Marketing Research · doi:10.1509/jmr.12.0106 · Cited by 774
“We then used Fleiss’ kappa coefficient (Fleiss 1971) to measure the agreement among the two independent human coders and the automated model.”
MethodSpotting fake reviewer groups in consumer reviews
Mukherjee, A., Liu, B., & Glance, N. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187863 · Cited by 773
“To study the feasibility of labeling groups and also the judging quality, we used Fleiss’ multi-rater kappa [10] to measure the judges’ agreements.”
MethodCONSIDERATIONS IN THE CHOICE OF INTEROBSERVER RELIABILITY ESTIMATES
Hartmann, D. P. (1977) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1977.10-103 · Cited by 762
Automatic "pipeline" analysis of 3-D MRI data for clinical trials: application to multiple sclerosis
Zijdenbos, A. P., Forghani, R., & Evans, A. C. (2002) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2002.806283 · Cited by 761
“For the analysis of the MRI simulation results, the classification accuracy (voxel-by-voxel agreement between the simulation label volume and the classified volume) was quantified using a similarity measure proposed by Dice [54], which was shown to be a special case of the kappa statistic [29], [55], [56], and referre…”
MethodA meta‐analytic investigation of the relationship between HRM bundles and firm performance
Subramony, M. (2009) · Human Resource Management · doi:10.1002/hrm.20315 · Cited by 734
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
Effects of alcohol on human aggression: An intergrative research review.
Bushman, B. J., & Cooper, H. M. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.3.341 · Cited by 710
“With one exception, the reliability coefficients (Fleiss, 1971) ranged from 0.”
BackgroundInfluentialThe ‘‘what’’and ‘‘how’’of case study rigor: three strategies based on published work
Gibbert, M., & Ruigrok, W. (2010) · Organizational Research Methods · doi:10.1177/1094428109351319 · Cited by 506
“Inter-rater reliability measured by Fleiss’ k, which provides a more robust measure than APPA because it takes into account the agreement occurring by chance, was 0.93 (Fleiss, 1971).”
MethodInfluentialTracking of body mass index in children in relation to overweight in adulthood
Guo, S. S., & Chumlea, W. C. (1999) · The American Journal of Clinical Nutrition · doi:10.1093/ajcn/70.1.145s · Cited by 704
From Text to Codings: Intercoder Reliability Assessment in Qualitative Content Analysis
Burla, L., Knierim, B., Barth, J., Liewald, K., Duetz, M., & Abel, T. (2008) · Nursing Research · doi:10.1097/01.nnr.0000313482.33917.7d · Cited by 699
“The consistency of codings of more than two coders can be assessed in a similar way with Fleiss’ (1971) kappa.”
MethodThe Anxiety Disorders Interview Schedule for Children
SILVERMAN, W. K., & NELLES, W. B. (1988) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-198811000-00019 · Cited by 688
(2011) · Cited by 675
Leadership behavior and employee well-being: An integrated review and a future research agenda
Inceoglu, I., Thomas, G., Chu, C., Plans, D., & Gerbasi, A. (2018) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.12.006 · Cited by 670
“Fleiss kappa was computed, a well-established measures of agreement for more than two raters (Fleiss, 1971), indicating high agreement for coding leadership behaviors (Fleiss kappa = 0.”
MethodInfluentialImplicit leadership theories: Content, structure, and generalizability
Offermann, L. R., Kennedy, J. K., & Wirtz, P. W. (1994) · The Leadership Quarterly · doi:10.1016/1048-9843(94)90005-1 · Cited by 633
Lee, W., Teckie, S., Wiesner, T., Ran, L., Prieto Granada, C. N., Lin, M., Zhu, S., Cao, Z., Liang, Y., Sboner, A., et al. (2014) · Nature Genetics · doi:10.1038/ng.3095 · Cited by 619
The Abbreviated Injury Scale and Injury Severity Score: Levels of Inter- and Intrarater Reliability
MacKenzie, E. J., Shapiro, S., & Eastham, J. N. (1985) · Medical Care · doi:10.1097/00005650-198506000-00008 · Cited by 616
A Re-analysis of the Reliability of Psychiatric Diagnosis
Spitzer, R. L., & Fleiss, J. L. (1974) · British Journal of Psychiatry · doi:10.1192/bjp.125.4.341 · Cited by 612
Self-citationPeer-assisted learning interventions with elementary school students: A meta-analytic review.
Rohrbeck, C. A., Ginsburg-Block, M. D., Fantuzzo, J. W., & Miller, T. R. (2003) · Journal of Educational Psychology · doi:10.1037/0022-0663.95.2.240 · Cited by 605
“Kappas and agreement percentages were above acceptable values (Fleiss, 1971; Sattler, 1992).”
BackgroundMarlovits, S., Singer, P., Zeller, P., Mandl, I., Haller, J., & Trattnig, S. (2006) · European Journal of Radiology · doi:10.1016/j.ejrad.2005.08.007 · Cited by 602
Wallace, C. A., Giannini, E. H., Huang, B., Itert, L., Ruperto, N., Childhood Arthritis Rheumatology Research Alliance (CARRA), Pediatric Rheumatology Collaborative Study Group (PRCSG), & Paediatric Rheumatology International Trials Organisation (PRINTO) (2011) · Arthritis Care & Research · doi:10.1002/acr.20497 · Cited by 574
“Intraphysician reliability was calculated using the unweighted kappa method as described by Fleiss (25).”
MethodA Taxonomy of Social Cues for Conversational Agents
Feine, J., Gnewuch, U., Morana, S., & Maedche, A. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.07.009 · Cited by 567
The high prevalence of pain in emergency medical care
Cordell, W. H., Keene, K. K., Giles, B. K., Jones, J. B., Jones, J. H., & Brizendine, E. J. (2002) · The American Journal of Emergency Medicine · doi:10.1053/ajem.2002.32643 · Cited by 571
External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)
Shea, B. J., Bouter, L. M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V. K., & Grimshaw, J. M. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001350 · Cited by 569
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563
“20 We used logistic regression (all the input variables were entered simultaneously) to examine the effect of the examination section, cognitive level, and dif fi culty index on ChatGPT ’ s answer accuracy.”
Fine-grained and accurate source code differencing
Falleri, J.-R., Morandat, F., Blanc, X., Martinez, M., & Monperrus, M. (2014) · Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/2642937.2642982 · Cited by 549
Fourney, D. R., Frangou, E. M., Ryken, T. C., DiPaola, C. P., Shaffrey, C. I., Berven, S. H., Bilsky, M. H., Harrop, J. S., Fehlings, M. G., Boriani, S., et al. (2011) · Journal of Clinical Oncology · doi:10.1200/jco.2010.34.3897 · Cited by 551
Towards a framework for software measurement validation
Kitchenham, B., Pfleeger, S. L., & Fenton, N. (1995) · IEEE Transactions on Software Engineering · doi:10.1109/32.489070 · Cited by 550
Integration and generalization of kappas for multiple raters
Conger, A. J. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.322 · Cited by 545
Optimization or Bricolage? Overcoming Resource Constraints in Global Social Entrepreneurship
Desa, G., & Basu, S. (2013) · Strategic Entrepreneurship Journal · doi:10.1002/sej.1150 · Cited by 522
Findings of the 2016 Conference on Machine Translation
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., et al. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2301 · Cited by 539
“481 according to Fleiss’ Kappa (Fleiss, 1971).”
Kappa coefficients in medical research
Chmura Kraemer, H., Periyakoil, V. S., & Noda, A. (2002) · Statistics in Medicine · doi:10.1002/sim.1180 · Cited by 527
Deep Learning for the Radiographic Detection of Periodontal Bone Loss
Krois, J., Ekert, T., Meinhold, L., Golla, T., Kharbot, B., Wittemeier, A., Dörfer, C., & Schwendicke, F. (2019) · Scientific Reports · doi:10.1038/s41598-019-44839-3 · Cited by 517
Wu, Y., Wu, W., Xing, C., Zhou, M., & Li, Z. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1046 · Cited by 506
Fiebach, J. B., Schellinger, P. D., Jansen, O., Meyer, M., Wilde, P., Bender, J., Schramm, P., Jüttler, E., Oehler, J., Hartmann, M., et al. (2002) · Stroke · doi:10.1161/01.str.0000026864.20339.cb · Cited by 507
Measuring Agreement between Two Judges on the Presence or Absence of a Trait
Fleiss, J. L. (1975) · Biometrics · doi:10.2307/2529549 · Cited by 501
Self-citationSharma, R., Yetton, P., & Crawford, J. (2009) · MIS Quarterly · doi:10.2307/20650305 · Cited by 496
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
Handbook of multimethod measurement in psychology.
(2006) · American Psychological Association eBooks · doi:10.1037/11383-000 · Cited by 481
The Kappa Statistic: A Second Look
Eugenio, B. D., & Glass, M. (2004) · Computational Linguistics · doi:10.1162/089120104773633402 · Cited by 489
A One-Way Components of Variance Model for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529465 · Cited by 478
“3) In this form it is apparent that the JpkkI are directly analogous to the components of kappa proposed in Fleiss (1971), and in particular, they represent agreement measures adjusted for expected agreement under independence and standardized by the appropriate variances.”
MethodInfluentialScheltens, P., Launer, L. J., Barkhof, F., Weinstein, H. C., & Gool, W. A. (1995) · Journal of Neurology · doi:10.1007/bf00868807 · Cited by 472
“According to Fleiss [ 6 ] a kappa value below 0.40 denotes poor agreement, between 0.40 and 0.75 means fair to good agreement, and a kappa value above 0.75 indicates excellent agreement.”
MethodInfluentialMeta-Analyses of Therapies for Postmenopausal Osteoporosis
Osteoporosis Methodology Group, & The Osteoporosis Research Advisory Group (2002) · Endocrine Reviews · doi:10.1210/edrv.23.4.0486 · Cited by 471
Cruchinho, P., López-Franco, M. D., Capelas, M. L., Almeida, S., Bennett, P. M., Miranda da Silva, M., Teixeira, G., Nunes, E., Lucas, P., & Gaspar, F. (2024) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s419714 · Cited by 469
“186,187 The minimum acceptable value of K is a coefficient of agreement of 0.60.”
InfluentialDesa, G. (2012) · Entrepreneurship Theory and Practice · doi:10.1111/j.1540-6520.2010.00430.x · Cited by 466
Rehm, J., Samokhvalov, A. V., Neuman, M. G., Room, R., Parry, C., Lönnroth, K., Patra, J., Poznyak, V., & Popova, S. (2009) · BMC Public Health · doi:10.1186/1471-2458-9-450 · Cited by 458
Longitudinal Study Comparing Sonographic and MRI Assessments of Acute and Healing Hamstring Injuries
Connell, D. A., Schneider-Kolsky, M. E., Hoving, J. L., Malara, F., Buchbinder, R., Koulouris, G., Burke, F., & Bass, C. (2004) · American Journal of Roentgenology · doi:10.2214/ajr.183.4.1830975 · Cited by 460
Moody, G., Cannings-John, R., Hood, K., Kemp, A., & Robling, M. (2018) · BMC Public Health · doi:10.1186/s12889-018-6044-y · Cited by 455
Ay, H., Benner, T., Murat Arsava, E., Furie, K. L., Singhal, A. B., Jensen, M. B., Ayata, C., Towfighi, A., Smith, E. E., Chong, J. Y., et al. (2007) · Stroke · doi:10.1161/strokeaha.107.490896 · Cited by 455
Reproducibility of histomorphologic diagnoses with special reference to the kappa statistic
SVANHOLM, H., STARKLINT, H., GUNDERSEN, H. J. G., FABRICIUS, J., BARLEBO, H., & OLSEN, S. (1989) · APMIS · doi:10.1111/j.1699-0463.1989.tb00464.x · Cited by 453
Reproducibility of a histopathologic grading system for adult soft tissue sarcoma
Coindre, J. M., Trojani, M., Contesso, G., David, M., Rouesse, J., Bui, N. B., Bodaert, A., De Mascarel, I., De Mascarel, A., & Goussot, J.-F. (1986) · Cancer · doi:10.1002/1097-0142(19860715)58:2<306::aid-cncr2820580216>3.0.co;2-7 · Cited by 449
Reproducibility of a histopathologic grading system for adult soft tissue sarcoma
Coindre, J., Trojani, M., Contesso, G., David, M., Rouëssé, J., Nguyen, B. B., Bodaert, A., de Mascarel, I., de Mascarel, A., & Goussot, J. (1986) · Cancer · doi:10.1002/1097-0142(19860715)58:2<222::aid-cncr2820580204>3.0.co;2-r · Cited by 448
Suboptimal reliability of liver biopsy evaluation has implications for randomized clinical trials
Davison, B. A., Harrison, S. A., Cotter, G., Alkhouri, N., Sanyal, A., Edwards, C., Colca, J. R., Iwashita, J., Koch, G. G., & Dittrich, H. C. (2020) · Journal of Hepatology · doi:10.1016/j.jhep.2020.06.025 · Cited by 448
THAT'S NOT MY JOB: DEVELOPING FLEXIBLE EMPLOYEE WORK.
Parker, S. K., Wall, T. D., & Jackson, P. R. (1997) · Academy of Management Journal · doi:10.2307/256952 · Cited by 447
“Z-scores resulting from testing the hypothesis of no-more-than-chance agreement (Fleiss, 1971) were all statistically significant, showing that assignment to categories was significantly better than chance.”
BackgroundClinical Interpretation and Implications of Whole-Genome Sequencing
Dewey, F. E., Grove, M. E., Pan, C., Goldstein, B. A., Bernstein, J. A., Chaib, H., Merker, J. D., Goldfeder, R. L., Enns, G. M., David, S. P., et al. (2014) · JAMA · doi:10.1001/jama.2014.1717 · Cited by 448
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.