Measuring nominal scale agreement among many raters.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Introduced the statistic kappa to measure nominal scale agreement between a fixed pair of raters. Kappa was generalized to the case where each of a sample of 30 patients was rated on a nominal scale by the same number of psychiatrist raters (n = 6), but where the raters rating 1 s were not necessarily the same as those rating another. Large sample standard errors were derived.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A kappa statistic was calculated to assess inter-rater reliability between the two coders and was found to be .78 (89.06% overall agreement, CI .63 to .94), which is above the recommended minimum levels of inter-rater reliability (Fleiss 1971).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Fleiss 1971
In the citing paper
“It also has advantages over other chance-corrected metrics like Cohen's Kappa (Cohen, 1960) and Fleiss' Kappa (Fleiss, 1971) with more flexibility supporting varying numbers of annotators, handling missing data, and accommodating different distance functions suited to the rating scale of each benchmark.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Fleiss, 1971
In the citing paper
“To assure inter-rater reliability, two coders assign codes to transcriptions independently.”
From Eco-Feedback for Food Waste Reduction in a Student Residence (2021) · cites this paper as Fleiss, 1971
In the citing paper
“The Fleiss Kappa (Fleiss, 1971) coefficient for round 1 annotation is 0.73, which is considered as substantial agreement.”
From Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022) · cites this paper as Fleiss, 1971
In the citing paper
“Inter-annotator agreement, measured by Fleiss' κ = 0.64, indicates substantial agreement (Fleiss, 1971).”
From Quantifying and Mitigating Self-Preference Bias of LLM Judges (2026) · cites this paper as Fleiss, 1971
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,520
Self-citationMOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 152
A STUDY OF PSYCHIATRIC DIAGNOSIS
Sandifer, M. G., Pettus, C. W., & Quade, D. (1964) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-196410000-00006 · Cited by 133
Sample selection by diagnosis in clinical drug evaluations
Sandifer, M. G., Fleiss, J. L., & Green, L. M. (1968) · Psychopharmacology · doi:10.1007/bf00404809 · Cited by 9
Self-citation
Cited by
Items 1,101–1,118 of 1,118
Development of a New Tool for Writing Research Key Information in Plain Language
Kurtz-Rossi, S., Okonkwo, I. A., Chen, Y., Dueñas, N., Bilodeau, T., Rushforth, A., & Klein, A. (2024) · HLRP: Health Literacy Research and Practice · doi:10.3928/24748307-20240218-01 · Cited by 6
“In Round 3 of testing, 9 items demonstrated moderate inter-rater agreement per Fleiss’ Kappa (average = .48) and 15 items demonstrated substantial agreement per Gwet’s AC1 (av - erage = .81).”
MethodReAct: A Review Comment Dataset for Actionability (and more)
Choudhary, G., Modani, N., & Maurya, N. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-91560-5_24 · Cited by 6
“In this paper, We use Fleiss’ kappa κ [5] as the measure of inter-annotator agreement.”
Methodvan Oest, R., & Girard, J. M. (2021) · Psychological Methods · doi:10.1037/met0000412 · Cited by 4
Explanatory argumentation in natural language for correct and incorrect medical diagnoses
Molinet, B., Marro, S., Cabrio, E., & Villata, S. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00306-1 · Cited by 4
“…agreement (IAA) has been calculated on an unseen shared subset of 10 clinical cases annotated by four annotators, obtaining a Fleiss’ kappa [81] of 0.70 for all of the annotated labels, 0.61 for Sign or Symptom , 0.94 for Location , 0.71 for Population Group , 0.66 for Finding , 0.96 for…”
MethodInfluentialSciClaimHunt: A Large Dataset for Evidence-based Scientific Claim Verification
Kumar, S., Sharma, A., Khincha, S. H., Shroff, G., Singh, S. R., & Mishra, R. (2025) · arXiv · doi:10.48550/arxiv.2502.10003 · Cited by 3
“Next, we measure inter-annotator agreement over the score assigned by annotators to one hundred claims for evaluation metrics Fluency , De-Contextualization , Atomicity , and Faithfulness of claims using Krippendorff α [32] and Fleiss kappa [33].”
A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations
Taniguchi, M., Ueda, Y., Taniguchi, T., & Ohkuma, T. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.436 · Cited by 3
“We use Fleiss’s kappa κ (Fleiss, 1971) as a measure of inter-annotator agreement.”
MethodA multimethod examination of hope and adolescent STEM career expectancies
Alexander, B. L., Janssen, J., Fraser, A. M., Bryce, C. I., & Fabes, R. A. (2022) · Journal of Adolescence · doi:10.1002/jad.12094 · Cited by 3
Schwartzman, J. M., Williams, Z. J., & Molnar, A. E. (2024) · Journal of Clinical Psychology in Medical Settings · doi:10.1007/s10880-023-09990-0 · Cited by 2
“To test agreement between raters (referring provider, parent, neuropsychologist) about the presence or absence of mental health and/or neurodevelopmental concerns, we examined three-way inter-rater agreement using Fleiss’s kappa coefficient (Fleiss, 1971).”
Özgüven Öztornaci, B., Karakul, A., Ardahan Akgül, E., Doğan, P., Yildirim Sari, H., & Başbakkal, Z. (2025) · Medicine · doi:10.1097/md.0000000000046222 · Cited by 1
“Since there were 4 observers in the evaluation of interrater reliability, Fleiss Kappa analysis [33] was used.”
The Illusion of Readiness in Health AI
Gu, Y., Fu, J., Liu, X., Valanarasu, J. M. J., Codella, N. C., Tan, R., Liu, Q., Jin, Y., Zhang, S., Wang, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.18234 · Cited by 1
“For every benchmark–axis pair, we report the median score and compute inter-annotator agreement using Fleiss’ κ [28].”
Self-Admitted GenAI Usage in Open-Source Software
Xiao, T., Fan, Y., Calefato, F., Treude, C., Kula, R. G., Hata, H., & Baltes, S. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3681886 · Cited by 1
“To assess inter-rater reliability, we computed Fleiss’ kappa [17] by applying bootstrap resampling methods with 1,000 iterations.”
Quadratically Weighted Agreement Coefficients: Interpretations and Connections
van Oest, R., & Moss, J. (2026) · Psychometrika · doi:10.1017/psy.2026.10085
Quantifying and Mitigating Self-Preference Bias of LLM Judges
Yang, J., Hu, Z., Qiu, C., Deng, Z., Jiao, X., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2604.22891
“64 , indicates substantial agreement [50].”
A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
Chen, J., Chen, W., Lin, Z., & Vong, C. M. (2026) · arXiv · doi:10.48550/arxiv.2608.24419
“Three independent annotators, no arbitration pass and no annotator seeing another’s labels, the label taken as the majority of the three, with Fleiss’ κ and raw pairwise agreement reported per class [8, 15].”
From Noise to Nuance: Enriching Subjective Data Annotation through Qualitative Analysis
Wan, R., Wang, H., Huang, T.-H. K., & Gao, J. (2025) · Proceedings of the Fourth Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP) · doi:10.18653/v1/2025.hcinlp-1.20
“Evaluating Quality In SDA, commonly used metrics are Fleiss’s kappa (Fleiss, 1971) (agreement among multiple annotators), Cohen’s kappa (Cohen, 1960) (agreement among two annotators), Krippendorff’s alpha (Krippendorff, 2011) (agreement among multiple annotators), percentage of disagreement,…”
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · arXiv · doi:10.48550/arxiv.2301.13298 · Cited by 3
“We measure Fleiss Kappa (Fleiss, 1971), Randolph Kappa (Randolph, 2005; Warrens, 2010), and the fraction of sentence pairs with total agreement.”
Method(2018) · DigitalCommons - Kennesaw State University (Kennesaw State University)
Metadata
- Authors
- Fleiss, J. L.
- Corresponding author
- Fleiss, Joseph L.
- Venue
- Psychological Bulletin
- Published
- 1971-11
- Volume
- 76
- Issue
- 5
- Pages
- 378-382
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W1975879668
- Cited by
- 8,733
- References
- 5
- Citations by year
- 2026: 666 · 2025: 517 · 2024: 510 · 2023: 655 · 2022: 561 · 2021: 624 · 2020: 606 · 2019: 430 · 2018: 435 · 2017: 411 · 2016: 345 · 2015: 332 · 2014: 356 · 2013: 296 · 2012: 292
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- interobserver agreement, multiple raters, categorical ratings, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.