The disagreeable behaviour of the kappa statistic: L. Flight and S. A. Julious
Abstract
It is often of interest to measure the agreement between a number of raters when an outcome is nominal or ordinal. The kappa statistic is used as a measure of agreement. The statistic is highly sensitive to the distribution of the marginal totals and can produce unreliable results. Other statistics such as the proportion of concordance, maximum attainable kappa and prevalence and bias adjusted kappa should be considered to indicate how well the kappa statistic represents agreement in the data. Each kappa should be considered and interpreted based on the context of the data being analysed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cohen's kappa has issues in the learning context causing unreliable results, including high sensitivity to the distribution of marginal totals.
“Despite its popularity, in the learning context there are a number of issues causing the kappa measure to produce unreliable results (for instance, its high sensitivity to the distribution of the marginal totals [73–75])”
From The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020) · cites this paper as [73]
The citation lists haven’t been fetched yet
References 11 · Cited by 77
- Authors
- Flight, L., & Julious, S. A.
- Affiliations
- University of Sheffield (GB)
- Corresponding author
- Julious, Steven A.
- Venue
- Pharmaceutical Statistics
- Publisher
- Wiley
- Published
- 2014-12-03
- Volume
- 14
- Issue
- 1
- Pages
- 74-78
- ISSN
- 1539-1604 (ISSN-L)
- OpenAlex
- W1540240814
- Cited by
- 77
- References
- 11
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 6 · 2023: 11 · 2022: 5 · 2021: 6 · 2020: 10 · 2019: 8 · 2018: 7 · 2017: 2 · 2016: 6 · 2015: 3
- FWCI
- 3.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Pharmaceutical Statistics (publishedVersion)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Funders
- Research Trainees Coordinating Centre
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- kappa statistic, interobserver agreement, marginal totals, ordinal data, nominal data
- MeSH terms
- Data Interpretation, Statistical, Humans, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
