Interpreting area under the receiver operating characteristic curve
Open accessAbstract
Clinicians often ask biostatisticians to provide a judgement on the quality of a prediction model. The prediction quality can be assessed with different performance criteria, with discriminative ability being a main issue: how well can we separate high-risk from low-risk patients? Discriminative ability is typically quantified by the area under the receiver operating characteristic curve (AUC) when we consider prediction of a binary event.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
High discriminatory ability is not sufficient to ensure positive effects from deploying prediction models in clinical practice.
“While AUC values above 0.7–0.8 are often considered clinically useful, labeling of AUC values (i.e., poor, moderate, good, and excellent) is arbitrary, and high discriminatory ability is not sufficient to ensure positive effects from deploying prediction models in clinical practice [57].”
From Factors associated with postpartum depression symptoms following antepartum hospitalization (2026) · cites this paper as [57]
Although discriminatory ability is essential for predictive AI, AUROC alone cannot be used to identify a model as good or useful.
“Nevertheless, although discriminatory ability is essential for predictive AI, AUROC alone cannot be used to identify a model as good or useful. 51”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Self-citationAlba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
The Threshold Approach to Clinical Decision Making
Pauker, S. G., & Kassirer, J. P. (1980) · New England Journal of Medicine · doi:10.1056/nejm198005153022003 · Cited by 1,371 · see citing works
Vickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163 · see citing works
Self-citationInterpreting Incremental Value of Markers Added to Risk Prediction Models
Pencina, M. J., D'Agostino, R. B., Pencina, K. M., Janssens, A. C. J. W., & Greenland, P. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws207 · Cited by 506 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
A review of statistical updating methods for clinical prediction models
Su, T.-L., Jaki, T., Hickey, G. L., Buchan, I., & Sperrin, M. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280215626466 · Cited by 147 · see citing works
Moons, K. G. M., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2015) · Advances in Anatomic Pathology · doi:10.1097/pap.0000000000000072 · Cited by 139 · see citing works
Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., et al. (2022) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.008487 · Cited by 60 · see citing works
Self-citationGordon, A. J., Govindarajan, P., Bennett, C., Matheson, L. W., Kohn, M. A., Camargo, C. A., & Kline, J. A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-054700 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Self-citationInstability of the AUROC of clinical prediction models
van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025) · Statistics in Medicine · doi:10.1002/sim.70011 · Cited by 19 · see citing works
Self-citationFactors associated with postpartum depression symptoms following antepartum hospitalization
Goulding, A. N., Palacios, D., Aras, S., Chen, H., Pasupuleti, S., Toscano, M., Cirino, N., Christie, I. C., Liu, Z., Miller, E. S., et al. (2026) · Pregnancy · doi:10.1002/pmf2.70319
- Authors
- de Hond, A. A. H., Steyerberg, E. W., & van Calster, B.
- Affiliations
- Leiden University Medical Center (NL)
- ORCID
- Steyerberg, Ewout W
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2022-12
- Volume
- 4
- Issue
- 12
- Pages
- e853-e855
- ISSN
- 2589-7500 (ISSN-L)
- OpenAlex
- W4306722800
- Cited by
- 503
- References
- 10
- Citations by year
- 2026: 123 · 2025: 184 · 2024: 153 · 2023: 41 · 2022: 2
- FWCI
- 70.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- The Lancet Digital Health (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Lirias (submittedVersion, other-oa)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- area under the receiver operating characteristic, AUC, discriminatory power, binary outcome prediction, risk stratification, clinical judgment
- MeSH terms
- ROC Curve, Area Under Curve
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
