Assessing the inter-rater agreement for ordinal data through weighted indexes
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Assessing the inter-rater agreement between observers, in the case of ordinal variables, is an important issue in both the statistical theory and biomedical applications. Typically, this problem has been dealt with the use of Cohen’s weighted kappa, which is a modification of the original kappa statistic, proposed for nominal variables in the case of two observers. Fleiss (1971) put forth a generalization of kappa in the case of multiple observers, but both Cohen’s and Fleiss’ kappa could have a paradoxical behavior, which may lead to a difficult interpretation of their magnitude. In this paper, a modification of Fleiss’ kappa, not affected by paradoxes, is proposed, and subsequently generalized to the case of ordinal variables. Monte Carlo simulations are used both to testing statistical hypotheses and to calculating percentile and bootstrap-t confidence intervals based on this statistic. The normal asymptotic distribution of the proposed statistic is demonstrated. Our results are applied to the classical Holmquist et al.’s (1967) dataset on the classification, by multiple observers, of carcinoma in situ of the uterine cervix. Finally, we generalize the use of s* to a bivariate case.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · cited 4×
“It has been shown that the use of the weighted kappa with M categories and weights in equation (4) is equivalent to use a kappa coefficient combined from M 1 embedded 2 2 tables.”
BackgroundMethodInfluentialWeighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“Fleiss (1971) put forth a generalization of kappa in the case of multiple observers, but both Cohen’s and Fleiss’ kappa could have a paradoxical behavior, which may lead to a difficult interpretation of their magnitude.”
BackgroundInterobserver agreement for the assessment of handicap in stroke patients
van Swieten, J. C., Koudstaal, P. J., Visser, M. C., Schouten, H. J., & van Gijn, J. (1988) · Stroke · doi:10.1161/01.str.19.5.604 · Cited by 6,512
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034
Bootstrap confidence intervals
DiCiccio, T. J., & Efron, B. (1996) · Statistical Science · doi:10.1214/ss/1032280214 · Cited by 2,286
Misinterpretation and misuse of the kappa statistic
MACLURE, M., & WILLETT, W. C. (1987) · American Journal of Epidemiology · doi:10.1093/aje/126.2.161 · Cited by 808
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
Elements of Large-Sample Theory
(1999) · Springer Texts in Statistics · doi:10.1007/b98855 · Cited by 677
Dependence of Weighted Kappa Coefficients on the Number of Categories
Brenner, H., & Kliebsch, U. (1996) · Epidemiology · doi:10.1097/00001648-199603000-00016 · Cited by 255
Schuster, C. (2004) · Educational and Psychological Measurement · doi:10.1177/0013164403260197 · Cited by 139
A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges
Janson, H., & Olsson, U. (2004) · Educational and Psychological Measurement · doi:10.1177/0013164403260195 · Cited by 81
Cited by
Fernández-Gómez, E., Martín-Salvador, A., Luque-Vara, T., Sánchez-Ojeda, M. A., Navarro-Prado, S., & Enrique-Mirón, C. (2020) · Nutrients · doi:10.3390/nu12041136 · Cited by 211
Unsupervised tip-mining from customer reviews
Zhu, D., Lappas, T., & Zhang, J. (2018) · Decision Support Systems · doi:10.1016/j.dss.2018.01.011 · Cited by 32
Hubert's multi‐rater kappa revisited
Martín Andrés, A., & Álvarez Hernández, M. (2020) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12167 · Cited by 16
Carl, M., Rudyk, E., Shapira, Y., Rusiewicz, H. L., & Icht, M. (2024) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2024_jslhr-24-00009 · Cited by 9
Metadata
- Authors
- Marasini, D., Quatto, P., & Ripamonti, E.
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2016-09-30
- Volume
- 25
- Issue
- 6
- Pages
- 2611-2633
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W1970423858
- Cited by
- 115
- References
- 13
- FWCI
- 1.92
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.