Implementing a General Framework for Assessing Interrater Agreement in Stata
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Despite its well-known weaknesses, researchers continuously choose the kappa coefficient (Cohen, 1960, Educational and Psychological Measurement 20: 37–46; Fleiss, 1971, Psychological Bulletin 76: 378–382) to quantify agreement among raters. Part of kappa's persistent popularity seems to arise from a lack of available alternative agreement coefficients in statistical software packages such as Stata. In this article, I review Gwet’s (2014, Handbook of Inter-Rater Reliability) recently developed framework of interrater agreement coefficients. This framework extends several agreement coefficients to handle any number of raters, any number of rating categories, any level of measurement, and missing values. I introduce the kappaetc command, which implements this framework in Stata.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 2×
“Fleiss called his coefficient kappa, giving the wrong impression that it is a generalized version of the coefficient proposed by Cohen (1960, 1968) before.”
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 5×
“359 for Fleiss’s kappa to 0.390 for Krippendorff’s alpha. All coefficients are statistically significant at the 5% level. Using the Landis and Koch (1977) scale, discussed in section 2.”
InfluentialAnswering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 2×
“The kalpha command (Klein 2014) adds circular and bipolar weights for partial agreement and implements a bootstrap algorithm, described by Hayes and Krippendorff (2007), to obtain confidence intervals.”
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034 · cited 1×
“A second paradox pointed out by Feinstein and Cicchetti (1990) is kappa’s dependency on the marginal distribution.”
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179 · cited 3×
“Researchers have also criticized kappa on various grounds and proposed alternative agreement coefficients (ACs) (Byrt, Bishop, and Carlin 1993; Feinstein and Cicchetti 1990; Gwet 2008a; Warrens 2012).”
InfluentialComputing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167 · cited 8×
“More recently, Gwet (2008a, 2014) proposes his AC that incorporates both the number of rating categories and the frequency with which they are used by the raters.”
InfluentialWongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-61 · Cited by 1,041
Integration and generalization of kappas for multiple raters
Conger, A. J. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.322 · Cited by 545
Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0200-9 · Cited by 377
Cited by
Mishra, S. (2020) · Educational Research Review · doi:10.1016/j.edurev.2019.100307 · Cited by 688
“2015) and the community-contributed software kappa etc. (Klein, 2018).”
Fully automated endoscopic disease activity assessment in ulcerative colitis
Yao, H., Najarian, K., Gryak, J., Bishu, S., Rice, M. D., Waljee, A. K., Wilkins, H. J., & Stidham, R. W. (2021) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2020.08.011 · Cited by 118
Lee, R. Y., Kross, E. K., Downey, L., Paul, S. R., Heywood, J., Nielsen, E. L., Okimoto, K., Brumback, L. C., Merel, S. E., Engelberg, R. A., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.5088 · Cited by 44
Metadata
- Authors
- Klein, D.
- Venue
- The Stata Journal: Promoting communications on statistics and Stata
- Publisher
- SAGE Publications
- Published
- 2018-12
- Volume
- 18
- Issue
- 4
- Pages
- 871-901
- ISSN
- 1536-867X (ISSN-L)
- 1536-8734
- OpenAlex
- W2911547067
- Cited by
- 158
- References
- 10
- FWCI
- 8.14
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Implementing a General Framework for Assessing Interrater Agreement in Stata. https://app.cochat.ai/papers/10.1177/1536867x1801800408
- The paper
- Klein, D. (2018). Implementing a General Framework for Assessing Interrater Agreement in Stata. The Stata Journal: Promoting communications on statistics and Stata. https://doi.org/10.1177/1536867x1801800408
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.