Limitations of the Odds Ratio in Gauging the Performance of a Diagnostic, Prognostic, or Screening Marker
The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
A marker strongly associated with outcome (or disease) is often assumed to be effective for classifying persons according to their current or future outcome. However, for this assumption to be true, the associated odds ratio must be of a magnitude rarely seen in epidemiologic studies. In this paper, an illustration of the relation between odds ratios and receiver operating characteristic curves shows, for example, that a marker with an odds ratio of as high as 3 is in fact a very poor classification tool. If a marker identifies 10% of controls as positive (false positives) and has an odds ratio of 3, then it will correctly identify only 25% of cases as positive (true positives). The authors illustrate that a single measure of association such as an odds ratio does not meaningfully describe a marker's ability to classify subjects. Appropriate statistical methods for assessing and reporting the classification power of a marker are described. In addition, the serious pitfalls of using more traditional methods based on parameters in logistic regression models are illustrated.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, for diagnostic, prognostic, or screening markers in epidemiologic studies, the limited magnitude of the OR considered meaningful (i.e., about or >70) is rarely discussed (36).”
From Misrepresentation and distortion of research in biomedical literature (2018) · cites this paper as [36]
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
A Gene-Expression Signature as a Predictor of Survival in Breast Cancer
van de Vijver, M. J., He, Y. D., van 't Veer, L. J., Dai, H., Hart, A. A. M., Voskuil, D. W., Schreiber, G. J., Peterse, J. L., Roberts, C., Marton, M. J., et al. (2002) · New England Journal of Medicine · doi:10.1056/nejmoa021967 · Cited by 6,488
Ridker, P. M., Hennekens, C. H., Buring, J. E., & Rifai, N. (2000) · New England Journal of Medicine · doi:10.1056/nejm200003233421202 · Cited by 5,850
Gail, M. H., Brinton, L. A., Byar, D. P., Corle, D. K., Green, S. B., Schairer, C., & Mulvihill, J. J. (1989) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/81.24.1879 · Cited by 3,334
The Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
Self-citationCarney, P. A., Miglioretti, D. L., Yankaskas, B. C., Kerlikowske, K., Rosenberg, R., Rutter, C. M., Geller, B. M., Abraham, L. A., Taplin, S. H., Dignan, M., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-3-200302040-00008 · Cited by 1,142
Association Between Myeloperoxidase Levels and Risk of Coronary Artery Disease
Zhang, R. (2001) · JAMA · doi:10.1001/jama.286.17.2136 · Cited by 918
Loss of IGF2 Imprinting: A Potential Marker of Colorectal Cancer Risk
Cui, H., Cruz-Correa, M., Giardiello, F. M., Hutcheon, D. F., Kafonek, D. R., Brandenburg, S., Wu, Y., He, X., Powe, N. R., & Feinberg, A. P. (2003) · Science · doi:10.1126/science.1080902 · Cited by 735
Improved confidence intervals for the difference between binomial proportions based on paired data
Newcombe, R. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19981130)17:22<2597::aid-sim940>3.0.co;2-g · Cited by 432
A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data
WIEAND, S., GAIL, M. H., JAMES, B. R., & JAMES, K. L. (1989) · Biometrika · doi:10.1093/biomet/76.3.585 · Cited by 355
Rhodes, D. R., Sanda, M. G., Otte, A. P., Chinnaiyan, A. M., & Rubin, M. A. (2003) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/95.9.661 · Cited by 271
Judging New Markers by Their Ability to Improve Predictive Accuracy
Kattan, M. W. (2003) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/95.9.634 · Cited by 265
Combining Several Screening Tests: Optimality of the Risk Score
McIntosh, M. W., & Pepe, M. S. (2002) · Biometrics · doi:10.1111/j.0006-341x.2002.00657.x · Cited by 251
Self-citationParodi, S., Izzotti, A., & Muselli, M. (2005) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/dji034 · Cited by 136
Boyko, E. J. (1994) · Medical Decision Making · doi:10.1177/0272989x9401400210 · Cited by 127
Liou, S.-H., Lung, J. C., Chen, Y.-H., Yang, T.-Y., Hsieh, L., Chen, C., & Wu, T. (1999) · PubMed · Cited by 120
Etzioni, R., PEPE, M. S., Longton, G., Hu, C., & Goodman, G. (1999) · Medical Decision Making · doi:10.1177/0272989x9901900303 · Cited by 106
Self-citationMarkers for early detection of cancer: Statistical guidelines for nested case-control studies
Baker, S. G., Kramer, B. S., & Srivastava, S. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-4 · Cited by 104
Identifying Combinations of Cancer Markers for Further Study as Triggers of Early Intervention
Baker, S. G. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.01082.x · Cited by 97
Semiparametric Regression for the Area Under the Receiver Operating Characteristic Curve
Dodd, L. E., & PEPE, M. S. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000198 · Cited by 94
Self-citationSensitivity and specificity for correlated observations
Smith, P. J., & Hadgu, A. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111108 · Cited by 83
Early Detection Cancer Research Network
Srivastava, S., & Kramer, B. S. (2000) · Laboratory Investigation · doi:10.1038/labinvest.3780122 · Cited by 55
Newcomb, P. A., & Trentham‐Dietz, A. (2003) · Cancer Causes & Control · doi:10.1023/a:1023066304473 · Cited by 43
Analysis of repeated markers used to predict progression of cancer
Emir, B., Wieand, S., Su, J. Q., & Cha, S. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19981130)17:22<2563::aid-sim952>3.0.co;2-o · Cited by 42
The use of risk factors in medical diagnosis: Opportunities and cautions
Boyko, E. J., & Alderman, B. W. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90068-z · Cited by 39
Hoque, A. M., Lippman, S. M., Boiko, I., Atkinson, E. N., Sneige, N., Şahin, A. A., Weber, D. M., Risin, S. A., Lagios, M. D., Schwarting, R., et al. (2001) · PubMed · Cited by 39
Lachenbruch, P. A. (1997) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)90013-1 · Cited by 11
Cited by
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
“Ware and Pepe show simple examples in which enormous odds ratios are required to meaningfully increase the AUC [ 19 , 21].”
BackgroundAssessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,112
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
Metadata
- Authors
- Pepe, M. S.
- Affiliations
- Fred Hutch Cancer Center (US)
- Corresponding author
- Pepe, M. S.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2004-05-01
- Volume
- 159
- Issue
- 9
- Pages
- 882-890
- ISSN
- 0002-9262 (ISSN-L)
- OpenAlex
- W2121514893
- Cited by
- 1,276
- References
- 27
- Citations by year
- 2026: 56 · 2025: 44 · 2024: 21 · 2023: 35 · 2022: 39 · 2021: 46 · 2020: 46 · 2019: 46 · 2018: 68 · 2017: 63 · 2016: 69 · 2015: 69 · 2014: 72 · 2013: 73 · 2012: 82
- FWCI
- 41.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- National Cancer Institute (U01CA86368, U01CA074794, P01 CA18029, U01CA086368, P01CA018029)
- National Institute of General Medical Sciences (R01 GM 54438, R01GM054438)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- odds ratio, diagnostic biomarkers, prognostic factors, screening marker, receiver operating characteristic, classification accuracy, false positives, true positives, logistic regression, association measures
- MeSH terms
- Classification, Data Interpretation, Statistical, Epidemiologic Methods, False Positive Reactions, Humans, Mass Screening, Predictive Value of Tests, Prognosis, Risk Factors, ROC Curve, Reproducibility of Results, Biomarkers, Discriminant Analysis, Logistic Models, Odds Ratio, Case-Control Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.