A Comparison of Reliability Coefficients for Ordinal Rating Scales
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Kappa coefficients are commonly used for quantifying reliability on a categorical scale, whereas correlation coefficients are commonly applied to assess reliability on an interval scale. Both types of coefficients can be used to assess the reliability of ordinal rating scales. In this study, we compare seven reliability coefficients for ordinal rating scales: the kappa coefficients included are Cohen’s kappa, linearly weighted kappa, and quadratically weighted kappa; the correlation coefficients included are intraclass correlation ICC(3,1), Pearson’s correlation, Spearman’s rho, and Kendall’s tau-b. The primary goal is to provide a thorough understanding of these coefficients such that the applied researcher can make a sensible choice for ordinal rating scales. A second aim is to find out whether the choice of the coefficient matters. We studied to what extent we reach the same conclusions about inter-rater reliability with different coefficients, and to what extent the coefficients measure agreement in a similar way, using analytic methods, and simulated and empirical data. Using analytical methods, it is shown that differences between quadratic kappa and the Pearson and intraclass correlations increase if agreement becomes larger. Differences between the three coefficients are generally small if differences between rater means and variances are small. Furthermore, using simulated and empirical data, it is shown that differences between all reliability coefficients tend to increase if agreement between the raters increases. Moreover, for the data in this study, the same conclusion about inter-rater reliability was reached in virtually all cases with the four correlation coefficients. In addition, using quadratically weighted kappa, we reached a similar conclusion as with any correlation coefficient a great number of times. Hence, for the data in this study, it does not really matter which of these five coefficients is used. Moreover, the four correlation coefficients and quadratically weighted kappa tend to measure agreement in a similar way: their values are very highly correlated for the data in this study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 4×
“…1993; Maclure and Willett 1987; Schouten 1986), while the weighted kappa coefficient (Cohen 1968) is widely used for quantifying agreement between ratings on an ordinal scale (Moradzadeh et al. 2017; Vanbelle 2016; Warrens 2012a, 2013, 2014; Vanbelle and Albert 2009; Crewson 2005; Cohen 1968).”
BackgroundMethodInfluentialMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 2×
“An extension of Scott’s pi to multiple raters is presented in Fleiss (1971).”
BackgroundForming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
Shiloach, M., Frencher, S. K., Steeger, J. E., Rowell, K. S., Bartzokis, K., Tomeh, M. G., Richards, K. E., Ko, C. Y., & Hall, B. L. (2010) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2009.09.031 · Cited by 1,306
de Winter, J. C. F., Gosling, S. D., & Potter, J. (2016) · Psychological Methods · doi:10.1037/met0000079 · Cited by 1,201
Misinterpretation and misuse of the kappa statistic
MACLURE, M., & WILLETT, W. C. (1987) · American Journal of Epidemiology · doi:10.1093/aje/126.2.161 · Cited by 808
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747 · cited 1×
“Multi-rater extensions of unweighted kappa are presented in Light (1971), Hubert (1977), Conger (1980), and Davies and Fleiss (1982).”
BackgroundWing, L., Leekam, S. R., Libby, S. J., Gould, J., & Larcombe, M. (2002) · Journal of Child Psychology and Psychiatry · doi:10.1111/1469-7610.00023 · Cited by 694
Integration and generalization of kappas for multiple raters
Conger, A. J. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.322 · Cited by 545 · cited 1×
“Multi-rater extensions of unweighted kappa are presented in Light (1971), Hubert (1977), Conger (1980), and Davies and Fleiss (1982).”
BackgroundDependence of Weighted Kappa Coefficients on the Number of Categories
Brenner, H., & Kliebsch, U. (1996) · Epidemiology · doi:10.1097/00001648-199603000-00016 · Cited by 255
Schuster, C. (2004) · Educational and Psychological Measurement · doi:10.1177/0013164403260197 · Cited by 139 · cited 6×
“If the ratings (scores) are labeled as 1, 2, 3, and so on, quadratic kappa is given by (Schuster 2004; Schuster and Smith 2005) κq = 2s12 s21 + s22 + nn−1 (m1 − m2)2 .”
BackgroundMethodInfluentialThe analysis of ordinal agreement data: beyond weighted kappa
Graham, P., & Jackson, R. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90173-x · Cited by 121
Corrected Zegers-ten Berge Coefficients Are Special Cases of Cohen’s Weighted Kappa
Warrens, M. J. (2014) · Journal of Classification · doi:10.1007/s00357-014-9156-9 · Cited by 18 · cited 4×
“However, the arbitrariness of the choice of weights is generally considered a weakness of the coefficient (Vanbelle 2016; Warrens 2012a, 2013, 2014; Vanbelle and Albert 2009; Crewson 2005; Maclure and Willett 1987).”
BackgroundMethodInfluentialSelf-citation
Cited by
AI-based differential diagnosis of dementia etiologies on multimodal data
Xue, C., Kowshik, S. S., Lteif, D., Puducheri, S., Jasodanand, V. H., Zhou, O. T., Walia, A. S., Guney, O. B., Zhang, J. D., Poésy, S., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03118-z · Cited by 182
Pack, A., Barrett, A., & Escalante, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100234 · Cited by 166
Human Gait phases recognition based on multi-source data fusion and BILSTM attention neural network
Zhan, H., Kou, J., Cao, Y., Guo, Q., Zhang, J., & Shi, Y. (2024) · Measurement · doi:10.1016/j.measurement.2024.115396 · Cited by 28
Metadata
- Authors
- de Raadt, A., Warrens, M. J., Bosker, R. J., & Kiers, H. A. L.
- Venue
- Journal of Classification
- Published
- 2021-04-22
- Volume
- 38
- Issue
- 3
- Pages
- 519-543
- ISSN
- 0176-4268 (ISSN-L)
- 1432-1343
- OpenAlex
- W3159595972
- Cited by
- 112
- References
- 16
- FWCI
- 8.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.