The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases
Abstract
Objectives Compare the area under the receiver operating characteristic curve (AUC) vs. the area under the precision-recall curve (AUPRC) in summarizing the performance of a diagnostic biomarker according to the disease prevalence.Study design and setting A simulation study was performed considering different sizes of diseased and nondiseased groups. Values of a biomarker were sampled with various variances and differences in mean values between the two groups. The AUCs and the AUPRCs were examined regarding their agreement and vs. the positive predictive value (PPV) and the negative predictive value (NPV) of the biomarker.Results With a disease prevalence of 50%, the AUC and the AUPRC showed high correlations with the PPV and the NPV (ρ > 0.95). With a prevalence of 1%, small PPV and AUPRC values (<0.2) but high AUC values (>0.9) were found. The AUPRC reflected better than the AUC the discriminant ability of the biomarker; it had a higher correlation with the PPV (ρ = 0.995 vs. 0.724; P < 0.001).Conclusion In uncommon and rare diseases, the AUPRC should be preferred to the AUC because it summarizes better the performance of a biomarker.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The precision-recall (PR) and receiver operating characteristic (ROC) curves of the four models were similar.
“The precision-recall (PR) and receiver operating characteristic (ROC) curves [68] of the four models were observed to be similar (Figure 7).”
From A Machine Learning-Based Quality Control Algorithm for Heavy Rainfall Using Multi-Source Data (2025) · cites this paper as [68]
Several scientists consider the PR curve more informative than the ROC curve, especially on imbalanced biological and medical datasets.
“Even if less common than the ROC curve, several scientists consider the PR curve more informative than the ROC curve, especially on imbalanced biological and medical datasets [48, 97, 98].”
From The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020) · cites this paper as [98]
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · see citing works
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · see citing works
Proceedings of the 23rd international conference on Machine learning
Cohen, W. W., & Moore, A. (2006) · Cited by 2,647 · see citing works
The Statistical Evaluation of Medical Tests for Classification and Prediction
PEPE, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,206 · see citing works
The Statistical Evaluation of Medical Tests for Classification and Prediction
PEPE, M. S. (2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,907 · see citing works
Evaluation Measures for Multi-class Subgroup Discovery
Abudawood, T., & Flach, P. A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04180-8_20 · Cited by 787 · see citing works
The damage-response framework of microbial pathogenesis
Casadevall, A., & Pirofski, L. (2003) · Nature Reviews Microbiology · doi:10.1038/nrmicro732 · Cited by 662 · see citing works
Uses and abuses of screening tests
Grimes, D. A., & Schulz, K. F. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07948-5 · Cited by 545 · see citing works
Therapeutic decision making: a cost-benefit analysis
Pauker, S. G., & Kassirer, J. P. (1975) · New England Journal of Medicine · doi:10.1056/nejm197507312930505 · Cited by 507 · see citing works
Probabilistic inference for solving discrete and continuous state Markov Decision Processes
Toussaint, M., & Storkey, A. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143963 · Cited by 496 · see citing works
Evaluation of diagnostic tests for infectious diseases: general principles
Banoo, S., Bell, D., Bossuyt, P. M., Herring, A. J., Mabey, D. C. W., Poole, F., Smith, P. G., Sriram, N., Wongsrichanalai, C., Linke, R., et al. (2006) · Nature Reviews Microbiology · doi:10.1038/nrmicro1523 · Cited by 454 · see citing works
Measurements of acute cerebral infarction: lesion size by computed tomography.
Brott, T. G., Marler, J. R., Olinger, C. P., Adams, H. P., Tomsick, T. A., Barsan, W. G., Biller, J., EBERLE, R. W., Hertzberg, V., & Walker, M. (1989) · Stroke · doi:10.1161/01.str.20.7.871 · Cited by 384 · see citing works
Liu, M., Wu, Y., Chen, Y., Sun, J., Zhao, Z., Chen, X.-W., Matheny, M. E., & Xu, H. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000699 · Cited by 291 · see citing works
Erratum: Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals
Boyd, K., Eng, K. H., & Page, C. D. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40994-3_55 · Cited by 155 · see citing works
The influence of referral patterns on the characteristics of diagnostic tests
AndréKnorrnerus, J., & Leffers, P. (1992) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(92)90155-g · Cited by 143 · see citing works
Evaluation of diagnostic tests for infectious diseases: general principles
Banoo, S., Bell, D., Bossuyt, P. M., et al. (2010) · Nature Reviews Microbiology · doi:10.1038/nrmicro1523z · Cited by 128 · see citing works
Machine Learning and Knowledge Discovery in Databases
Calders, T., Esposito, F., Hüllermeier, E., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44845-8 · Cited by 58 · see citing works
Bouts, M. J. R. J., Tiebosch, I. A., van der Toorn, A., Viergever, M. A., Wu, O., & Dijkhuizen, R. M. (2013) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2013.51 · Cited by 51 · see citing works
Musculoskeletal conditions associated with Wilson’s disease
Quemeneur, A.-S., Trocello, J., Ea, H., et al. (2011) · Best Practice & Research Clinical Rheumatology · doi:10.1016/j.berh.2011.10.021 · Cited by 36 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · see citing works
Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging
Arun, N., Gaw, N., Singh, P., Chang, K., Aggarwal, M., Chen, B., Hoebel, K., Gupta, S., Patel, J., Gidwani, M., et al. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021200267 · Cited by 342 · see citing works
An algorithm based on deep learning for predicting in-hospital cardiac arrest
Kwon, J., Lee, Y., Lee, Y., Lee, S., & Park, J. (2018) · Journal of the American Heart Association · doi:10.1161/jaha.118.008678 · Cited by 299 · see citing works
Sabanayagam, C., Xu, D., Ting, D. S. W., Nusinovici, S., Banu, R., Hamzah, H., Lim, C., Tham, Y.-C., Cheung, C. Y., Tai, E. S., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30063-7 · Cited by 279 · see citing works
Machine learning model to predict mental health crises from electronic health records
Garriga, R., Mas, J., Abraha, S., Nolan, J., Harrison, O., Tadros, G., & Matic, A. (2022) · Nature Medicine · doi:10.1038/s41591-022-01811-5 · Cited by 229 · see citing works
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Peeking into a black box, the fairness and generalizability of a MIMIC-III benchmarking model.
Röösli, E., Bozkurt, S., & Hernandez-Boussard, T. (2022) · Scientific Data · doi:10.1038/s41597-021-01110-7 · Cited by 73 · see citing works
Uyeda, A. M., Curtis, J. R., Engelberg, R. A., Brumback, L. C., Guo, Y., Sibley, J., Lober, W. B., Cohen, T., Torrence, J., Heywood, J., et al. (2022) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2022.02.006 · Cited by 19 · see citing works
On Missingness Features in Machine Learning Models for Critical Care: Observational Study
Singh, J., Sato, M., & Ohkuma, T. (2021) · JMIR Medical Informatics · doi:10.2196/25022 · Cited by 17 · see citing works
Validation and comparison of triage-based screening strategies for sepsis
Rahmati, K., Brown, S. M., Bledsoe, J. R., Passey, P., Taillac, P. P., Youngquist, S. T., Samore, M. M., Hough, C. L., & Peltan, I. D. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.08.037 · Cited by 4 · see citing works
Generative Medical Event Models Improve with Scale
Waxler, S., Blazek, P., White, D., Sneider, D., Chung, K., Nagarathnam, M., Williams, P., Voeller, H., Wong, K., Swanhorst, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.12104 · Cited by 4 · see citing works
Ali, W., Williams, J., Xiong, B., Zou, J., & Daneshjou, R. (2025) · JID Innovations · doi:10.1016/j.xjidi.2025.100362 · Cited by 4 · see citing works
A Machine Learning-Based Quality Control Algorithm for Heavy Rainfall Using Multi-Source Data
Sun, H., Zhou, Q., Shi, L., Li, C., Qin, S., Yao, D., Xu, M., Huang, Y., Hu, Q., & Guan, Y. (2025) · Remote Sensing · doi:10.3390/rs17243976 · Cited by 4 · see citing works
- Authors
- Ozenne, B., Subtil, F., & Maucort-Boulch, D.
- Affiliations
- Lyon 1 Université (FR)
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2015-08
- Volume
- 68
- Issue
- 8
- Pages
- 855-859
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2154039449
- Cited by
- 379
- References
- 19
- Citations by year
- 2026: 31 · 2025: 56 · 2024: 61 · 2023: 58 · 2022: 54 · 2021: 46 · 2020: 44 · 2019: 10 · 2018: 10 · 2017: 3 · 2016: 3 · 2015: 2
- FWCI
- 5.34
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- HAL (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- precision-recall curve, area under the receiver operating characteristic, rare diseases, disease prevalence, diagnostic biomarkers, positive predictive value, negative predictive value, biomarker discrimination, diagnostic accuracy
- MeSH terms
- Humans, Predictive Value of Tests, ROC Curve, Sensitivity and Specificity, Biomarkers, Prevalence, Area Under Curve, Rare Diseases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
