ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models
Open accessAbstract
OBJECTIVES: Receiver operating characteristic (ROC) curves show how well a risk prediction model discriminates between patients with and without a condition. We aim to investigate how ROC curves are presented in the literature and discuss and illustrate their potential limitations. STUDY DESIGN AND SETTING: We conducted a pragmatic literature review of contemporary publications that externally validated clinical prediction models. We illustrated limitations of ROC curves using a testicular cancer case study and simulated data. RESULTS: Of 86 identified prediction modeling studies, 52 (60%) presented ROC curves without thresholds and one (1%) presented an ROC curve with only a few thresholds. We illustrate that ROC curves in their standard form withhold threshold information have an unstable shape even for the same area under the curve (AUC) and are problematic for comparing model performance conditional on threshold. We compare ROC curves with classification plots, which show sensitivity and specificity conditional on risk thresholds. CONCLUSION: ROC curves do not offer more information than the AUC to indicate discriminative ability. To assess the model's performance for decision-making, results should be provided conditional on risk thresholds. Therefore, if discriminatory ability must be visualized, classification plots are attractive.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For binary outcomes, the C-statistic is equal to area under the receiver operating characteristic curve (AUROC).
“For binary outcomes, the C-statistic is equal to area under the receiver operating characteristic curve (AUROC). 37,38”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [38]
A limitation of ROC and PR curves is that the thresholds are not easily visible.
“A limitation of these plots is that the thresholds are not easily visible (figure 1).”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · see citing works
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Self-citationMoons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Self-citationEvaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490 · see citing works
ROCR: visualizing classifier performance in R
Sing, T., Sander, O., Beerenwinkel, N., & Lengauer, T. (2005) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti623 · Cited by 3,172 · see citing works
Time‐Dependent ROC Curves for Censored Survival Data and a Diagnostic Marker
Heagerty, P. J., Lumley, T., & PEPE, M. S. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00337.x · Cited by 2,870 · see citing works
The Statistical Evaluation of Medical Tests for Classification and Prediction
PEPE, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,206 · see citing works
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,123 · see citing works
The Statistical Evaluation of Medical Tests for Classification and Prediction
PEPE, M. S. (2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,907 · see citing works
Perkins, N. J., & Schisterman, E. F. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj063 · Cited by 1,753 · see citing works
Uno, H., Cai, T., Pencina, M. J., D'Agostino, R. B., & Wei, L. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4154 · Cited by 1,535 · see citing works
Self-citationSurvival Model Predictive Accuracy and ROC Curves
Heagerty, P. J., & Zheng, Y. (2005) · Biometrics · doi:10.1111/j.0006-341x.2005.030814.x · Cited by 1,471 · see citing works
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,458 · see citing works
Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators
CALSTER, B. V., Wynants, L., Verbeek, J. F. M., Verbakel, J. Y. J., Christodoulou, E., Vickers, A. J., Roobol, M. J., & Steyerberg, E. W. (2018) · European Urology · doi:10.1016/j.eururo.2018.08.038 · Cited by 1,289 · see citing works
Self-citationVickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163 · see citing works
Self-citationCriteria for Evaluation of Novel Markers of Cardiovascular Risk
Hlatky, M., Greenland, P., Arnett, D., Ballantyne, C., Criqui, M., Elkind, M., Go, A., Harrell, F. E. J., Hong, Y.-L., Howard, B., et al. (2009) · Circulation · doi:10.1161/circulationaha.109.192278 · Cited by 1,153 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
Self-citationTime-dependent ROC curve analysis in medical research: current methods and applications
Kamarudin, A. N., Cox, T. F., & Kolamunnage‐Dona, R. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0332-6 · Cited by 839 · see citing works
The use of receiver operating characteristic curves in biomedical informatics
Lasko, T. A., Bhagwat, J. G., Zou, K. H., & Lucila Ohno-machado, A. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.02.008 · Cited by 821 · see citing works
Kerr, K. F., Brown, M. D., Zhu, K., & Janes, H. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.65.5654 · Cited by 697 · see citing works
Therapeutic decision making: a cost-benefit analysis
Pauker, S. G., & Kassirer, J. P. (1975) · New England Journal of Medicine · doi:10.1056/nejm197507312930505 · Cited by 507 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/l15-5093-2 · Cited by 493 · see citing works
External correspondence: Decompositions of the mean probability score
Yates, J. (1982) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(82)90237-9 · Cited by 466 · see citing works
ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions
Obuchowski, N. A., Lieber, M. L., & Wians, F. H. (2004) · Clinical Chemistry · doi:10.1373/clinchem.2004.031823 · Cited by 358 · see citing works
Misuse of DeLong test to compare AUCs for nested models
Demler, O., Pencina, M. J., & D’Agostino, R. B. (2012) · Statistics in Medicine · doi:10.1002/sim.5328 · Cited by 313 · see citing works
Estimation of time‐dependent area under the ROC curve for long‐term risk prediction
Chambless, L. E., & Diao, G. (2005) · Statistics in Medicine · doi:10.1002/sim.2299 · Cited by 309 · see citing works
The Youden Index and the Optimal Cut‐Point Corrected for Measurement Error
Perkins, N. J., & Schisterman, E. F. (2005) · Biometrical Journal · doi:10.1002/bimj.200410133 · Cited by 289 · see citing works
Interpreting diagnostic accuracy studies for patient care
Mallett, S., Halligan, S., Thompson, M., Collins, P. G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e3999 · Cited by 249 · see citing works
Newcombe, R. G. (2005) · Statistics in Medicine · doi:10.1002/sim.2323 · Cited by 159 · see citing works
Using Relative Utility Curves to Evaluate Risk Prediction
Baker, S. G., Cook, N. R., Vickers, A., & Kramer, B. S. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00592.x · Cited by 142 · see citing works
Qin, G., & Hotilovac, L. (2007) · Statistical Methods in Medical Research · doi:10.1177/0962280207087173 · Cited by 126 · see citing works
Comparing risk prediction models
Collins, G. S., & Moons, K. G. M. (2012) · BMJ · doi:10.1136/bmj.e3186 · Cited by 110 · see citing works
Steyerberg, E. W., Gerl, A., Fosså, S. D., Sleijfer, D. T., de Wit, R., Kirkels, W. J., Schmeller, N. T., Clemm, C., Habbema, J. D. F., & Keizer, H. J. (1998) · Journal of Clinical Oncology · doi:10.1200/jco.1998.16.1.269 · Cited by 105 · see citing works
Self-citationPEPE, M. S., Janes, H., Li, C. I., Bossuyt, P. M., Feng, Z., & Hilden, J. (2016) · Clinical Chemistry · doi:10.1373/clinchem.2015.252163 · Cited by 87 · see citing works
Vergouwe, Y., Steyerberg, E. W., Foster, R. S., Sleijfer, D. T., Fosså, S. D., Gerl, A., de Wit, R., Roberts, J. T., & Habbema, J. D. F. (2006) · European Urology · doi:10.1016/j.eururo.2006.06.047 · Cited by 85 · see citing works
Self-citationPeirce, Youden, and Receiver Operating Characteristic Curves
Baker, S. G., & Kramer, B. S. (2007) · The American Statistician · doi:10.1198/000313007x247643 · Cited by 80 · see citing works
Leening, M. J. G., Steyerberg, E. W., CALSTER, B. V., D’Agostino, R. B., & Pencina, M. J. (2014) · Statistics in Medicine · doi:10.1002/sim.6133 · Cited by 55 · see citing works
Self-citationMethods for Evaluating Prediction Performance of Biomarkers and Tests
PEPE, M. S., & Janes, H. (2013) · Lecture notes in statistics · doi:10.1007/978-1-4614-8981-8_7 · Cited by 44 · see citing works
Risk Assessment and Evaluation of Predictions
Lee, M. T., Gail, M. H., Pfeiffer, R. M., Satten, G. A., Cai, T., & Gandy, A. (2013) · Lecture notes in statistics · doi:10.1007/978-1-4614-8981-8 · Cited by 40 · see citing works
Steyerberg, E. W., Vedder, M. M., Leening, M. J. G., Postmus, D., D’Agostino, R. B., CALSTER, B. V., & Pencina, M. J. (2014) · Biometrical Journal · doi:10.1002/bimj.201300260 · Cited by 29 · see citing works
Self-citationDetermining Decision Limits for New Biomarkers: Clinical and Statistical Considerations
Morrow, D. A., & Cook, N. R. (2010) · Clinical Chemistry · doi:10.1373/clinchem.2010.155879 · Cited by 23 · see citing works
Statistical graphics in action: making better sense of the ROC curve
Althouse, A. D. (2016) · International Journal of Cardiology · doi:10.1016/j.ijcard.2016.04.026 · Cited by 20 · see citing works
Risk Prediction for Individuals
CALSTER, B. V., Steyerberg, E. W., & Harrell, F. H. (2015) · JAMA · doi:10.1001/jama.2015.12215 · Cited by 16 · see citing works
Self-citationCALSTER, B. V., Steyerberg, E. W., D’Agostino, R. B., & Pencina, M. J. (2013) · Medical Decision Making · doi:10.1177/0272989x13513654 · Cited by 9 · see citing works
Self-citationEfficient nonparametric confidence bands for receiver operating-characteristic curves
Martínez‐Camblor, P., Pérez‐Fernández, S., & Corral, N. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280216672490 · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Van Calster B, Steyerberg EW, Harrell FH. Risk prediction for indi- viduals. JAMA 2015;314:1875.
Cited by
Sorted by citations
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R., Archer, L., Snell, K., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L., & Collins, G. S. (2024) · British medical journal · doi:10.1136/bmj-2023-074820 · Cited by 515 · see citing works
Rosati, D., Palmieri, M., Brunelli, G., Morrione, A., Iannelli, F., Frullanti, E., & Giordano, A. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.02.018 · Cited by 196 · see citing works
Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, P. G. S., & Barnett, A. G. (2023) · BMC Medicine · doi:10.1186/s12916-023-03048-6 · Cited by 149 · see citing works
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
Self-citationBaquero, J. P. H., Chumillas, M. R., Perez, A. G., Escámez, P. S. F., Samper, A. A., Iguaz, A., Esnoz, A., & Gómez, A. P. (2013) · Foods · doi:10.3390/foods · Cited by 101 · see citing works
Aktar, S., Ahamad, M. M., Rashed-Al-Mahfuz, M., Azad, A., Uddin, S., Kamal, A. H. M., Alyami, S. A., Lin, P., Islam, S. M. S., Quinn, J. M. W., et al. (2021) · JMIR Medical Informatics · doi:10.2196/25884 · Cited by 84 · see citing works
Hằng, H. T., Mallick, J., Alqadhi, S. D., Bindajam, A. A., & Abdo, H. G. (2024) · Environmental Technology & Innovation · doi:10.1016/j.eti.2024.103655 · Cited by 74 · see citing works
Fazel, S., Burghart, M., Fanshawe, T. R., Gil, S. D., Monahan, J. T., & Yu, R. (2022) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2022.101902 · Cited by 66 · see citing works
Yu, M.-Y., & Son, Y. (2024) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvae031 · Cited by 34 · see citing works
Safari, A., Adibi, A., Sin, D. D., Lee, T. Y., Ho, J. K., Sadatsafavi, M., & IMPACT Study Team (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101574 · Cited by 34 · see citing works
Patarata, L., Fernandes, L., Silva, J. A., & Fraqueza, M. J. (2022) · Foods · doi:10.3390/foods11030444 · Cited by 32 · see citing works
Methodologic Issues Specific to Prediction Model Development and Evaluation
Jin, Y., & Kattan, M. W. (2023) · CHEST Journal · doi:10.1016/j.chest.2023.06.038 · Cited by 31 · see citing works
Huynh, T.-T. M., Dale, E., Falk, R. S., Hellebust, T. P., Astrup, G. L., Malinen, E., Edin, N. F. J., Bjordal, K., Herlofson, B. B., Kiserud, C. E., et al. (2023) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2023.110044 · Cited by 31 · see citing works
Zhao, Z., Cai, Q., Zhang, P., He, B., Peng, X., Tu, G., Peng, W., Wang, L., Yu, F., & Wang, X. (2021) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2021.657087 · Cited by 28 · see citing works
Yue, R., Jiang, G., Jin, X., He, Q., & Xie, P. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3350147 · Cited by 27 · see citing works
Jiang, H., Yang, F., Zhang, X., Fang, H., Qiu, T., Li, Y., & Peng, A. (2024) · Microbial Pathogenesis · doi:10.1016/j.micpath.2024.106888 · Cited by 25 · see citing works
Mbuya-Bienge, C., Pashayan, N., Kazemali, C. D., Lapointe, J., Simard, J., & Nabi, H. (2023) · Cancers · doi:10.3390/cancers15225380 · Cited by 24 · see citing works
Liu, Y., Chang, Y., Jiang, X., Mei, H., Cao, Y., Wu, D., Xie, R., Jiang, W., Vasquez, E., Wu, Y., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26219 · Cited by 25 · see citing works
Liu, J., Li, M., Li, R., Shalamzari, M. J., Ren, Y., & Silakhori, E. (2025) · Land · doi:10.3390/land14020337 · Cited by 24 · see citing works
Kuil, S. D., Hidad, S., Fischer, J., Harting, J., Hertogh, C. M. P. M., Prins, J. M., de Jong, M. D., van Leth, F., & Schneeberger, C. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa1709 · Cited by 23 · see citing works
Prediction models for functional status in community dwelling older adults: a systematic review
Grootven, B. V., & van Achterberg, T. (2022) · BMC Geriatrics · doi:10.1186/s12877-022-03156-7 · Cited by 22 · see citing works
Zuercher, P., Moser, A., de Guadiana‐Romualdo, L. G., Llewelyn, M., Graf, R., Reding, T., Eggimann, P., Que, Y., & Prazak, J. (2023) · Infection · doi:10.1007/s15010-023-02093-w · Cited by 22 · see citing works
Soeterik, T. F. W., Hueting, T. A., Israël, B., van Melick, H. H. E., Dijksman, L. M., Stomps, S. P., Biesma, D. H., Koffijberg, H., Sedelaar, J. P. M., Witjes, J. A., et al. (2021) · British Journal of Urology · doi:10.1111/bju.15376 · Cited by 21 · see citing works
The Misuse of AUC: What High Impact Risk Assessment Gets Wrong
Kwegyir-Aggrey, K., Gerchick, M., Mohan, M., Horowitz, A., & Venkatasubramanian, S. (2023) · Conference on Fairness, Accountability and Transparency · doi:10.1145/3593013.3594100 · Cited by 22 · see citing works
Hao, X., Zheng, D., Khan, M., Wang, L., Hämäläinen, T., Cong, F., Xu, H., & Song, K. (2023) · Diagnostics · doi:10.3390/diagnostics13040612 · Cited by 17 · see citing works
Naye, F., Décary, S., Houle, C., LeBlanc, A., Cook, C. E., Dugas, M., Skidmore, B., & Tousignant‐Laflamme, Y. (2023) · Physical Therapy · doi:10.1093/ptj/pzad021 · Cited by 16 · see citing works
Hueting, T. A., van Maaren, M. C., Hendriks, M. P., Koffijberg, H., & Siesling, S. (2023) · The Breast · doi:10.1016/j.breast.2023.04.003 · Cited by 16 · see citing works
Popat, A., Yadav, S., Patel, S. K., Baddevolu, S., Adusumilli, S., Dasari, N. R., Sundarasetty, M., Anand, S., Sankar, J., & Jagtap, Y. G. (2023) · Cureus · doi:10.7759/cureus.50395 · Cited by 15 · see citing works
Yang, Y., Li, D., Liu, R., Hu, Y., Chen, S., Wu, S., & Tian, Y. (2023) · Hypertension Research · doi:10.1038/s41440-023-01313-y · Cited by 16 · see citing works
A deep learning hybrid ensemble fusion for chest radiograph classification
Sultana, S., Hussain, S. S., Hashmani, M. A., Ahmad, J., & Zubair, M. (2021) · Neural Network World · doi:10.14311/nnw.2021.31.010 · Cited by 13 · see citing works
Spatial mapping Zataria multiflora using different machine-learning algorithms
Edalat, M., Dastres, E., Jahangiri, E., Moayedi, G., Zamani, A., Pourghasemi, H. R., & Tiefenbacher, J. P. (2022) · CATENA · doi:10.1016/j.catena.2021.106007 · Cited by 13 · see citing works
Predicting retail customers' distress in the finance industry: An early warning system approach
Beltman, J., Machado, M., & Osterrieder, J. R. (2024) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2024.104101 · Cited by 12 · see citing works
Cao, K., Yeung, J., Arafat, Y., Choi, C., Wei, M. Y.-K., Chan, S. T. F., Lee, M., Baird, P. N., & Yeung, J. M. C. (2023) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-023-05227-7 · Cited by 12 · see citing works
Comparison and interpretability of machine learning models to predict severity of chest injury
Kulshrestha, S., Dligach, D., Joyce, C., Gonzalez, R., O’Rourke, A. P., Glazer, J. M., Stey, A. M., Kruser, J. M., Churpek, M. M., & Afshar, M. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab015 · Cited by 12 · see citing works
Qian, W., Li, S., & Lu, J. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3148993 · Cited by 12 · see citing works
Beenhakker, L., Wijlens, K. A. E., Witteveen, A., Heins, M. J., Korevaar, J. C., de Ligt, K. M., Bode, C., Vollenbroek-Hutten, M. M. R., & Siesling, S. (2023) · Journal of Cancer Survivorship · doi:10.1007/s11764-023-01491-1 · Cited by 10 · see citing works
Ishøi, L., Thorborg, K., Kallemose, T., Kemp, J. L., Reiman, M. P., Nielsen, M. F., & Hölmich, P. (2023) · British Journal of Sports Medicine · doi:10.1136/bjsports-2022-105534 · Cited by 9 · see citing works
Dastres, E., Sarvestani, G. S., Edalat, M., & Pourghasemi, H. R. (2025) · Weed Science · doi:10.1017/wsc.2025.5 · Cited by 9 · see citing works
Song, H., Qin, W.-B., Yang, F., Tang, W.-Z., & He, G. (2024) · BMC Cardiovascular Disorders · doi:10.1186/s12872-024-04357-1 · Cited by 8 · see citing works
Xie, X., Tiggelaar, S., Guo, J., Wang, M., Vandersluis, S., & Ungar, W. J. (2022) · Medical Decision Making · doi:10.1177/0272989x221089268 · Cited by 9 · see citing works
Losciale, J. M., Truong, L. K., Ward, P., Collins, P. G. S., & Bullock, G. S. (2024) · International Journal of Sports Physical Therapy · doi:10.26603/001c.122644 · Cited by 7 · see citing works
Hojati, M., Naderi, R., Edalat, M., & Pourghasemi, H. R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0322442 · Cited by 7 · see citing works
Papagiannaki, K., Kotroni, V., Lagouvardos, K., Bezes, A., Vafeiadis, V., Messini, I., Kroustallis, E., & Totos, I. (2022) · Water · doi:10.3390/w14060994 · Cited by 7 · see citing works
Sun, M., Li, Y., Hu, G., Yang, H., Zhang, G., Miao, L., Zhang, Y., Li, L., & Liu, J. (2022) · Journal of Separation Science · doi:10.1002/jssc.202200611 · Cited by 7 · see citing works
Prediction begins with diagnosis: Estimating seizure recurrence risk in the First Seizure Clinic
Ooi, S., Tailby, C., Nagino, N., Carney, P. W., Jackson, G. D., & Vaughan, D. N. (2024) · Seizure · doi:10.1016/j.seizure.2024.09.013 · Cited by 6 · see citing works
ROC curves for clinical prediction models part 3. The ROC plot: a picture that needs a 1000 words
CALSTER, B. V., Wynants, L., Collins, P. G. S., Verbakel, J. Y. J., & Steyerberg, E. W. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.05.037 · Cited by 6 · see citing works
Self-citationZhan, Y., Ripley-Gonzalez, J. W., Liu, Y., Gao, Y., Tang, J., & Gao, S. (2025) · Diabetology & Metabolic Syndrome · doi:10.1186/s13098-025-02016-9 · Cited by 5 · see citing works
Bujang, M. A. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1808675 · Cited by 7 · see citing works
Yao, C., Huang, L., Wang, M., Mao, D., Wang, M., Zheng, J., Long, F., Huang, J., Liu, X., Zhang, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-47012-z · Cited by 5 · see citing works
Tang, R., Jin, Y., Xu, K., Wu, L., Chen, X., Guo, Y., Li, G., & Li, J. (2025) · Brain Imaging and Behavior · doi:10.1007/s11682-024-00964-w · Cited by 5 · see citing works
Olivera-Salazar, R., Cabañas, G. S., Vega-Clemente, L., Alonso-Martín, D., Megías, V. M. C., Volward, P., García‐Olmo, D., & García‐Arranz, M. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25126783 · Cited by 5 · see citing works
Gérardin, P., Maillard, O., Bruneau, L., Accot, F., Legrand, F., Poubeau, P., MANAQUIN, R., Andry, F., Bertolotti, A., & Levin, C. (2021) · Travel Medicine and Infectious Disease · doi:10.1016/j.tmaid.2021.102232 · Cited by 5 · see citing works
Wang, X., Ye, C., Li, E., Xu, L., Lin, W., & Chen, G. (2022) · Journal of Cellular Biochemistry · doi:10.1002/jcb.30357 · Cited by 5 · see citing works
Xue, X., Wang, X., Ye, C., Gao, M., Peng, L., Yu, K., & Chen, G. (2024) · Chemical Research in Chinese Universities · doi:10.1007/s40242-024-4150-1 · Cited by 4 · see citing works
Geysels, A., Garofalo, G., Timmerman, S., Ceusters, J., Buonomo, F., Fischerová, D., Testa, A. C., Moro, F., Sladkevicius, P., Jokubkiene, L., et al. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-01215-x · Cited by 4 · see citing works
Fasham, T., Attal, J., Gouze, H., Abdel-Gawad, S., Dursun, E., & François, P. (2024) · Dental Materials Journal · doi:10.4012/dmj.2024-053 · Cited by 4 · see citing works
Hybrid Sensing for Fatigue Detection Using Wearables and RF
Taylor, W., Hill, D., Adam, R., Cooper, J. M., Abbasi, Q. H., & Imran, M. A. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3404637 · Cited by 4 · see citing works
Ahmed, N. N., Bhat, T. K., & Powar, O. S. (2024) · Systems Science & Control Engineering · doi:10.1080/21642583.2024.2427028 · Cited by 4 · see citing works
Do, X.-H., Le, M. T., Nguyen, T. H., Le, T. T., Nguyen, X., Mai, T. B., Hoang, T. M. N., & Than, U. T. T. (2023) · Journal of Hepatocellular Carcinoma · doi:10.2147/jhc.s409649 · Cited by 4 · see citing works
Shi, S., Peng, G., Luo, L., & Li, D. (2023) · Translational Cancer Research · doi:10.21037/tcr-23-1229 · Cited by 4 · see citing works
Medical Relevance, State-of-the-Art and Perspectives of “Sweet Metacode” in Liquid Biopsy Approaches
Pinkeová, A., Košútová, N., Jáné, E., Lorencová, L., Bertóková, A., Bertók, T., & Tkáč, J. (2024) · Diagnostics · doi:10.3390/diagnostics14070713 · Cited by 4 · see citing works
Park, J., Miley, E. N., Zhong, X., & Gray, C. F. (2025) · Knee Surgery and Related Research · doi:10.1186/s43019-025-00289-y · Cited by 3 · see citing works
Kamal, V. K., Agrawal, D. K., Kumar, A., Radhakrishnan, R., Ponnaiah, M., Ramaiya, D., Thiruvalluvan, A., & Sivaram, A. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-096275 · Cited by 3 · see citing works
Tourkmani, A. M., Alharbi, T., Al-Ghamdi, A., Youzghadli, I. M., Alosaimi, F. S., & Azzam, A. Y. (2025) · Diabetes Obesity and Metabolism · doi:10.1111/dom.70385 · Cited by 3 · see citing works
Tateiwa, T., Ishida, T., Takahashi, Y., Shinozaki, T., Masaoka, T., Shishido, T., Nishida, J., & Yamamoto, K. (2021) · Journal of Orthopaedic Science · doi:10.1016/j.jos.2021.04.014 · Cited by 3 · see citing works
Miao, X., Han, Y., Wu, Z., Jin, X., Niu, M., Zhao, Q., & Lü, X. (2024) · International Journal of COPD · doi:10.2147/copd.s455171 · Cited by 3 · see citing works
Alptekin, C., Wohlgemut, J., Perkins, Z., Marsh, W., Tai, N., & Yet, B. (2024) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2024.105702 · Cited by 2 · see citing works
Orimaye, S. O., & Schmidtke, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1362230 · Cited by 3 · see citing works
Mbatyoti, S., Sumner, P. D., Kalumba, A. M., Owolabi, S. T., & Belle, J. A. (2024) · Cuaternario y geomorfología: Revista de la Sociedad Española de Geomorfología y Asociación Española para el Estudio del Cuaternario · doi:10.17735/cyg.v38i3-4.101016 · Cited by 3 · see citing works
Risk scores in anaesthesia: the future is hard to predict
Drayton, D. J., Ayres, M., Relton, S. D., Sperrin, M., & Hall, M. (2022) · BJA Open · doi:10.1016/j.bjao.2022.100027 · Cited by 3 · see citing works
Zhao, W., Huang, L., Zhu, M., Li, X., Liu, X., Zhang, W., Shao, C., & Li, Y. (2025) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2025.1703044 · Cited by 2 · see citing works
Lu, A., Srikanth, V. K., Westworth, S. E., Baey, Y.-G., Moran, C., Beare, R. J., Siostrom, K., Andrew, N. E., & Collyer, T. A. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag048 · Cited by 2 · see citing works
Li, Y., Cui, J., Liu, Y., Chen, K., Huang, L., & Liu, Y. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.748256 · Cited by 2 · see citing works
Miranda, E., Kumbangsila, M., Aryuni, M., Richard, R., Zakiyyah, A. Y., & Sano, A. V. D. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-0248-4_11 · Cited by 2 · see citing works
Naesager, A. H. D., Damgaard, S. N., Rozing, M. P., Siersma, V., Møller, A., & Jensen, K. T. (2024) · BMC Psychiatry · doi:10.1186/s12888-024-05743-x · Cited by 2 · see citing works
Padullés, B., López-Aladid, R., Ingelmo‐Torres, M., Roldán, F. L., Martínez, C., Juez, J., Izquierdo, L., Mengual, L., & Alcaraz, A. (2025) · Cancers · doi:10.3390/cancers17162619 · Cited by 2 · see citing works
ROC Analyses Based on Measuring Evidence Using the Relative Belief Ratio
Al‐Labadi, L., Evans, M. J., & Liang, Q. (2022) · Entropy · doi:10.3390/e24121710 · Cited by 2 · see citing works
Predicting Retail Customers' Distress: Early Warning Systems and Machine Learning ApplicationsPreprint
Beltman, J., Osterrieder, J. R., & Machado, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4730470 · Cited by 2 · see citing works
Hu, A., Qin, H., Wu, S., Zhao, X., Li, Y. Q., Chen, F., & Liu, T. (2024) · Brazilian Journal of Medical and Biological Research · doi:10.1590/1414-431x2024e13359 · Cited by 1 · see citing works
Prediction of Range of Motion in Patients After Total Knee Arthroplasty by Shear Wave Elastography
Kim, M.-W., & Lee, D.-H. (2025) · Bioengineering · doi:10.3390/bioengineering12101009 · Cited by 1 · see citing works
Sarvestani, G. S., Edalat, M., Dastres, E., Naderi, R., & Pourghasemi, H. R. (2025) · Industrial Crops and Products · doi:10.1016/j.indcrop.2025.121974 · Cited by 1 · see citing works
Simon, J. E., Kraiński, Ł., Karlinski, M. A., & Niewada, M. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14207327 · Cited by 1 · see citing works
Data enhancement to improve CNN accuracy for early detection of colorectal cancer
Athiyah, U., Rustad, S., Soeleman, M. A., Muljono, M., Akrom, M., & Konoralma, R. A. (2025) · Array · doi:10.1016/j.array.2025.100581 · Cited by 1 · see citing works
Long, Q., Wang, Y., Axelin, A., Zheng, F., Cao, Z., Li, X., & Guo, J. (2025) · BMC Psychology · doi:10.1186/s40359-025-03809-3 · Cited by 1 · see citing works
Pesqué, D., de Frutos, F. J. O., Navarro‐Triviño, F. J., Sanz‐Sánchez, T., Zaragoza‐Ninet, V., Guijarro, S. C., Miquel‐Miquel, J., Salvador, J. F. S., González‐Pérez, R., González, I. R., et al. (2026) · Contact Dermatitis · doi:10.1111/cod.70089 · Cited by 1 · see citing works
Arzhangi, A., Partani, S., & Mehrjerdi, L. F. (2026) · Environmental Advances · doi:10.1016/j.envadv.2026.100694 · Cited by 1 · see citing works
Vereijken, F., Reps, J., Rijnbeek, P. R., & Williams, R. D. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag075 · Cited by 1 · see citing works
Zhang, H., Shi, Q., Xiao, C., Wang, J., & (21935), N. W. (2026) · Habitat International · doi:10.1016/j.habitatint.2026.103865 · Cited by 1 · see citing works
Razavi-Termeh, S. V., Sadeghi-Niaraki, A., Pourzangbar, A., Kişi, Ö., Hussain, J., & Choi, S.-M. (2026) · Water Resources Management · doi:10.1007/s11269-026-04827-5 · Cited by 1 · see citing works
DATA MINING APPLICATION FOR FINANCIAL DECISION OPTIMIZATION AT RISK
KOÇOĞLU, E., & Ersöz, F. (2021) · International Journal of 3D Printing Technologies and Digital Industry · doi:10.46519/ij3dptdi.950062 · Cited by 1 · see citing works
The correlation between the inner canthal distance and maxillary mesiodens in children
Tadano, M., Matsunaga, Y., Saito, K., Suzuki, Y., Nakamura, T., Hoshikawa, S., Chiba, M., Hino, R., Maruya, Y., Fukumoto, E., et al. (2023) · Pediatric Dental Journal · doi:10.1016/j.pdj.2023.04.001 · Cited by 1 · see citing works
Wang, Y., Yang, N. Y., Zhang, J., Han, W.-Y., & Li, X. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e32391 · Cited by 1 · see citing works
Human activity recognition based on machine learning and FMCW radar
Faraj, A. A., Al-Nakkash, A. H., & Wadday, A. G. (2024) · AIP conference proceedings · doi:10.1063/5.0236567 · Cited by 1 · see citing works
Li, H., Wang, Q., Wang, H., Li, L., Wang, X., Wu, T., Wang, C., Qian, Y., Wang, X., Xia, Y., et al. (2024) · Horticulturae · doi:10.3390/horticulturae10121321 · Cited by 1 · see citing works
PIWI-interacting RNA biomarkers in gastrointestinal disease
Huang, X.-Y., Chen, S., Wang, Z., Lu, Y.-S., Liu, C.-T., & Chen, S.-Z. (2025) · Clinica Chimica Acta · doi:10.1016/j.cca.2025.120182 · Cited by 1 · see citing works
Machine Learning Techniques in Bipolar Disorder
Rebouças, D. B., Barreto, P., Noronha, L. T., Roza, T. H., & Passos, I. C. (2025) · Springer eBooks · doi:10.1007/978-3-031-85519-1_40 · Cited by 2 · see citing works
Bülow, E. (2020) · Gothenburg University Publications Electronic Archive (Gothenburg University) · Cited by 1 · see citing works
Kim, Y. J., van den Broek, T., Brouwer-Brolsma, E. M., Feskens, E. J. M., Bouwman, J., & Kwon, O. (2025) · Research Square · doi:10.21203/rs.3.rs-6960301/v1 · Cited by 1 · see citing works
Rafiq, M., Mazzocato, P., Guttmann, C., Spaak, J., & Savage, C. (2022) · doi:10.2196/preprints.39847
- Authors
- Verbakel, J. Y., Steyerberg, E. W., Uno, H., De Cock, B., Wynants, L., Collins, G. S., & Van Calster, B.
- Affiliations
- University of Oxford (GB)
- Leiden University Medical Center (NL)
- Dana-Farber Cancer Institute (US)
- KU Leuven (BE)
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2020-10
- Volume
- 126
- Pages
- 207-216
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W3045258609
- Cited by
- 138
- References
- 48
- Citations by year
- 2026: 26 · 2025: 34 · 2024: 29 · 2023: 21 · 2022: 15 · 2021: 9 · 2020: 3 · 2013: 1
- FWCI
- 8.46
- Type
- journal article
- Language
- en
- Update notices
- erratum (2021-02-01)
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by-nc-nd)
- Lirias (KU Leuven) (submittedVersion, cc-by-nc-nd)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by-nc-nd)
- Funders
- Cancer Research UK
- Patient-Centered Outcomes Research Institute
- Fonds Wetenschappelijk Onderzoek
- NIHR Oxford Biomedical Research Centre
- Seventh Framework Programme
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Keywords
- ROC curve, area under the curve, clinical prediction model, risk prediction model, sensitivity and specificity, risk thresholds, model discrimination, external validation, testicular cancer
- MeSH terms
- Clinical Decision Rules, Computer Simulation, Decision Making, Humans, Lymph Nodes, Male, Models, Theoretical, Predictive Value of Tests, Retroperitoneal Neoplasms, ROC Curve, Sensitivity and Specificity, Testicular Neoplasms, Models, Statistical, Risk Assessment, Area Under Curve
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review