Bias, prevalence and kappa
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Since the introduction of Cohen's kappa as a chance-adjusted measure of agreement between two observers, several "paradoxes" in its interpretation have been pointed out. The difficulties occur because kappa not only measures agreement but is also affected in complex ways by the presence of bias between observers and by the distributions of data across the categories that are used ("prevalence"). In this paper, new indices that provide independent measures of bias and prevalence, as well as of observed agreement, are defined and a simple formula is derived that expresses kappa in terms of these three indices. When comparisons are made between agreement studies it can be misleading to report kappa values alone, and it is recommended that researchers also include quantitative indicators of bias and prevalence.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The maximum possible agreement is often less than unity because kappa values are influenced by the prevalence of the outcome under study and by the amount of bias present between examiners(12, 13).”
From Examiner training and reliability in two randomized clinical trials of adult dental caries. (2011) · cites this paper as [12]
In the citing paper
“we conducted a sensitivity analysis by calculating the prevalence-adjusted and bias-adjusted κ (PABAκ).19”
From Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models (2024) · cites this paper as [19]
In the citing paper
“The inter-rater reliability between the two reviewers was very strong ( PABAK = 0.919; Byrt et al., 1993).”
From Is the Putative Mirror Neuron System Associated with Empathy? A Systematic Review and Meta-Analysis (2021) · cites this paper as Byrt et al., 1993
In the citing paper
“The inter-rater reliability between the two reviewers was strong (PABAK = 0.818; Byrt et al., 1993).”
From Is the Putative Mirror Neuron System Associated with Empathy? A Systematic Review and Meta-Analysis (2021) · cites this paper as Byrt et al., 1993
In the citing paper
“Regarding validation screening, the degree of interrater agreement between the first and fourth author was very strong for title/ abstract screening (PABAK = 0.94; Byrt et al., 1993) and strong for full-text eligibility screening (PABAK = 0.77; Byrt et al., 1993).”
From Distress tolerance across substance use, eating, and borderline personality disorders: A meta-analysis (2022) · cites this paper as Byrt et al., 1993
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
High agreement but low kappa: II. Resolving the paradoxes
Cicchetti, D. V., & Feinstein, A. R. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90159-m · Cited by 1,838
Statistical Methods for Rates and Proportions.
Plewis, I., & Fleiss, J. L. (1982) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2982107 · Cited by 1,731
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,570
Misinterpretation and misuse of the kappa statistic
MACLURE, M., & WILLETT, W. C. (1987) · American Journal of Epidemiology · doi:10.1093/aje/126.2.161 · Cited by 808
A rEAPPRAISAL OF THE KAPPA COEFFICIENT
Thompson, W. D., & Walter, S. D. (1988) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(88)90031-5 · Cited by 497
Cited by
Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., Eccles, M. P., Cane, J., & Wood, C. E. (2013) · Annals of Behavioral Medicine · doi:10.1007/s12160-013-9486-6 · Cited by 7,629
“For both rounds, each intervention description was coded independently by two team members, and inter-rater agreement by BCT was assessed using kappas adjusted for prevalence and bias effects [35, 36].”
MethodALLOUCHE, O., TSOAR, A., & KADMON, R. (2006) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2006.01214.x · Cited by 5,172
“the prevalence and bias adjusted kappa, PABAK, proposed by Byrt, Bishop & Carlin 1993; the adjusted kappa coefficient, κ′ described by Hoehler 2000).”
BackgroundMethodInfluentialThe kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884
“R syntax will compute kappa statistics based on Cohen (1960), Siegel and Castellan (1988), and Byrt et al. (1993). deviates from the mean and represents the interaction between subject deviation and coder deviation.”
BackgroundMethodHuang, D. W., Sherman, B. T., Tan, Q., Collins, J. R., Alvord, W. G., Roayaei, J., Stephens, R., Baseler, M. W., Lane, H. C., & Lempicki, R. A. (2007) · Genome Biology · doi:10.1186/gb-2007-8-9-r183 · Cited by 2,479
“Then, kappa statistics, a chance-cor-rected measure of co-occurrence between two sets of categorized data, is adopted to statistically measure the annotation co-occurrence of any given gene pairs [30,31].”
Method
Metadata
- Authors
- Byrt, T., Bishop, J., & Carlin, J. B.
- Affiliations
- Royal Children's Hospital (AU)
- Corresponding author
- Byrt, Ted
- ORCID
- Carlin, John B.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 1993-05
- Volume
- 46
- Issue
- 5
- Pages
- 423-429
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2070285512
- Cited by
- 2,179
- References
- 16
- Citations by year
- 2026: 197 · 2025: 110 · 2024: 106 · 2023: 120 · 2022: 117 · 2021: 122 · 2020: 116 · 2019: 142 · 2018: 114 · 2017: 129 · 2016: 104 · 2015: 106 · 2014: 98 · 2013: 76 · 2012: 74
- FWCI
- 17.46
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- Cohen's kappa, interobserver agreement, observer bias, prevalence, chance-corrected agreement, categorical data
- MeSH terms
- Data Interpretation, Statistical, Humans, Mathematics, Observer Variation, Prevalence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bias, prevalence and kappa. https://app.cochat.ai/papers/10.1016/0895-4356(93)90018-v
- The paper
- Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology. https://doi.org/10.1016/0895-4356(93)90018-v
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.