The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
The analysis of ordinal agreement data: beyond weighted kappa
Abstract
The weighted kappa statistic has been used as an agreement index for ordinal data. Using data on the comparability of primary and proxy respondent reports of alcohol drinking frequency we show that the value of weighted kappa can be sensitive to the choice of weights. The distinction between association and agreement is clarified and it is shown that in some respects weighted kappa behaves more like a measure of association than an index of agreement. In particular, it is demonstrated that the weighted kappa statistic is not always sensitive to differences in the observed proportion in exact agreement and that high values of weighted kappa can be observed even when the level of agreement is low. We illustrate the use of statistical models in the analysis of epidemiologic agreement data and conclude that modelling ordinal agreement data produces insights which cannot be obtained through the use of weighted kappa statistics.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Weighted kappa is most appropriately used in the assessment of reliability for ordered classifications but this measure should be interpreted with caution not only in light of the weighting issue but because it behaves more like a measure of association than an index of agreement (16).”
From Examiner training and reliability in two randomized clinical trials of adult dental caries. (2011) · cites this paper as [16]
Citations
Cites
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Hapeman, D. E. (1991) · Technometrics · doi:10.1080/00401706.1991.10484817 · Cited by 9,373
Statistical Methods for Rates and Proportions.
Everitt, B. S., & Fleiss, J. L. (1981) · Biometrics · doi:10.2307/2530193 · Cited by 8,990
Measurement in Medicine: The Analysis of Method Comparison Studies
Altman, D. G., & Bland, J. M. (1983) · The Statistician · doi:10.2307/2987937 · Cited by 4,300
Fleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
Misinterpretation and misuse of the kappa statistic
MACLURE, M., & WILLETT, W. C. (1987) · American Journal of Epidemiology · doi:10.1093/aje/126.2.161 · Cited by 808
Patient-proxy response comparability on measures of patient health and functional status
Magaziner, J., Simonsick, E. M., Kashner, T. M., & Hebel, J. R. (1988) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(88)90076-5 · Cited by 339
2 x 2 Kappa Coefficients: Measures of Agreement or Association
Bloch, D. A., & Kraemer, H. C. (1989) · Biometrics · doi:10.2307/2532052 · Cited by 291
Alcohol consumption and risk of coronary heart disease.
Jackson, R., Scragg, R., & Beaglehole, R. (1991) · BMJ · doi:10.1136/bmj.303.6796.211 · Cited by 258
Self-citationPROXY RESPONDENTS IN EPIDEMIOLOGIC RESEARCH
NELSON, L. M., LONGSTRETH, W. T., KOEPSELL, T. D., & VAN BELLE, G. (1990) · Epidemiologic Reviews · doi:10.1093/oxfordjournals.epirev.a036063 · Cited by 205
Cited by
Statistical Techniques For Comparing Measurers And Methods Of Measurement: A Critical Review
Ludbrook, J. (2002) · Clinical and Experimental Pharmacology and Physiology · doi:10.1046/j.1440-1681.2002.03686.x · Cited by 455
Reliability of Self-rated Health in US Adults
Zajacova, A., & Dowd, J. B. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwr204 · Cited by 358
“Details about the calculation and interpretation of kappa and weighted kappa coefficients are available elsewhere (17, 18).”
BackgroundDependence of Weighted Kappa Coefficients on the Number of Categories
Brenner, H., & Kliebsch, U. (1996) · Epidemiology · doi:10.1097/00001648-199603000-00016 · Cited by 255
Statistical methods for assessing agreement for ordinal data
Jakobsson, U., & Westergren, A. (2005) · Scandinavian Journal of Caring Sciences · doi:10.1111/j.1471-6712.2005.00368.x · Cited by 203
“So, even if the exact agreement changes, the Kappa value is unchanged if the row and column marginal are unchanged (13).”
Backgroundvan der Wulp, I., van Baar, M. E., & Schrijvers, A. J. P. (2008) · Emergency Medicine Journal · doi:10.1136/emj.2007.055228 · Cited by 181
Metadata
- Authors
- Graham, P., & Jackson, R.
- Affiliations
- University of Auckland (NZ)
- ORCID
- Graham, Patrick
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 1993-09
- Volume
- 46
- Issue
- 9
- Pages
- 1055-1062
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1976386573
- Cited by
- 121
- References
- 25
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 1 · 2023: 1 · 2022: 2 · 2021: 5 · 2020: 4 · 2019: 3 · 2018: 6 · 2017: 1 · 2016: 6 · 2015: 3 · 2014: 10 · 2013: 5 · 2012: 4
- FWCI
- 1.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- weighted kappa, measurement agreement
- MeSH terms
- Alcohol Drinking, Epidemiologic Methods, Humans, Models, Statistical, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The analysis of ordinal agreement data: beyond weighted kappa. https://app.cochat.ai/papers/10.1016/0895-4356(93)90173-x
- The paper
- Graham, P., & Jackson, R. (1993). The analysis of ordinal agreement data: beyond weighted kappa. Journal of Clinical Epidemiology. https://doi.org/10.1016/0895-4356(93)90173-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.