Cited by 138Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsImbalanced Data Classification Techniques

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.

ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models

Open access

Verbakel, Jan Y., Steyerberg, Ewout W., Uno, Hajime, De Cock, Bavo, Wynants, Laure, Collins, Gary S., Van Calster, Ben·2020·Journal of Clinical Epidemiology

DOI 10.1016/j.jclinepi.2020.01.028

Open this paper in CoChat — free account Open PDF

Abstract

OBJECTIVES: Receiver operating characteristic (ROC) curves show how well a risk prediction model discriminates between patients with and without a condition. We aim to investigate how ROC curves are presented in the literature and discuss and illustrate their potential limitations. STUDY DESIGN AND SETTING: We conducted a pragmatic literature review of contemporary publications that externally validated clinical prediction models. We illustrated limitations of ROC curves using a testicular cancer case study and simulated data. RESULTS: Of 86 identified prediction modeling studies, 52 (60%) presented ROC curves without thresholds and one (1%) presented an ROC curve with only a few thresholds. We illustrate that ROC curves in their standard form withhold threshold information have an unstable shape even for the same area under the curve (AUC) and are problematic for comparing model performance conditional on threshold. We compare ROC curves with classification plots, which show sensitivity and specificity conditional on risk thresholds. CONCLUSION: ROC curves do not offer more information than the AUC to indicate discriminative ability. To assess the model's performance for decision-making, results should be provided conditional on risk thresholds. Therefore, if discriminatory ability must be visualized, classification plots are attractive.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2020.01.028

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Self-citation
Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citation
Evaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490
ROCR: visualizing classifier performance in R
Sing, T., Sander, O., Beerenwinkel, N., & Lengauer, T. (2005) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti623 · Cited by 3,172
Time‐Dependent ROC Curves for Censored Survival Data and a Diagnostic Marker
Heagerty, P. J., Lumley, T., & PEPE, M. S. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00337.x · Cited by 2,870
The Statistical Evaluation of Medical Tests for Classification and Prediction
PEPE, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,206

Show all 48 references

Sorted by citations

Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 565
Differential gene expression analysis pipelines and bioinformatic tools for the identification of specific biomarkers: A review
Rosati, D., Palmieri, M., Brunelli, G., Morrione, A., Iannelli, F., Frullanti, E., & Giordano, A. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.02.018 · Cited by 216
Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, P. G. S., & Barnett, A. G. (2023) · BMC Medicine · doi:10.1186/s12916-023-03048-6 · Cited by 149
Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112
Self-citation
Foods
Baquero, J. P. H., Chumillas, M. R., Perez, A. G., Escámez, P. S. F., Samper, A. A., Iguaz, A., Esnoz, A., & Gómez, A. P. (2013) · Foods · doi:10.3390/foods · Cited by 101
Machine Learning Approach to Predicting COVID-19 Disease Severity Based on Clinical Blood Test Data: Statistical Analysis and Model Development
Aktar, S., Ahamad, M. M., Rashed-Al-Mahfuz, M., Azad, A., Uddin, S., Kamal, A. H. M., Alyami, S. A., Lin, P., Islam, S. M. S., Quinn, J. M. W., et al. (2021) · JMIR Medical Informatics · doi:10.2196/25884 · Cited by 84
Exploring forest fire susceptibility and management strategies in Western Himalaya: Integrating ensemble machine learning and explainable AI for accurate prediction and comprehensive analysis
Hằng, H. T., Mallick, J., Alqadhi, S. D., Bindajam, A. A., & Abdo, H. G. (2024) · Environmental Technology & Innovation · doi:10.1016/j.eti.2024.103655 · Cited by 74
The predictive performance of criminal risk assessment tools used at sentencing: Systematic review of validation studies
Fazel, S., Burghart, M., Fanshawe, T. R., Gil, S. D., Monahan, J. T., & Yu, R. (2022) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2022.101902 · Cited by 80
Machine learning–based 30-day readmission prediction models for patients with heart failure: a systematic review
Yu, M.-Y., & Son, Y. (2024) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvae031 · Cited by 40
ACCEPT 2·0: Recalibrating and Externally Validating the Acute COPD Exacerbation Prediction Tool (ACCEPT)
Safari, A., Adibi, A., Sin, D. D., Lee, T. Y., Ho, J. K., Sadatsafavi, M., & IMPACT Study Team (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101574 · Cited by 35
The Risk of Salt Reduction in Dry-Cured Sausage Assessed by the Influence on Water Activity and the Survival of Salmonella
Patarata, L., Fernandes, L., Silva, J. A., & Fraqueza, M. J. (2022) · Foods · doi:10.3390/foods11030444 · Cited by 32
Methodologic Issues Specific to Prediction Model Development and Evaluation
Jin, Y., & Kattan, M. W. (2023) · CHEST Journal · doi:10.1016/j.chest.2023.06.038 · Cited by 31
Radiation-induced long-term dysphagia in survivors of head and neck cancer and association with dose-volume parameters
Huynh, T.-T. M., Dale, E., Falk, R. S., Hellebust, T. P., Astrup, G. L., Malinen, E., Edin, N. F. J., Bjordal, K., Herlofson, B. B., Kiserud, C. E., et al. (2023) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2023.110044 · Cited by 31
N6-Methyladenosine RNA Methylation Regulator-Related Alternative Splicing (AS) Gene Signature Predicts Non–Small Cell Lung Cancer Prognosis
Zhao, Z., Cai, Q., Zhang, P., He, B., Peng, X., Tu, G., Peng, W., Wang, L., Yu, F., & Wang, X. (2021) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2021.657087 · Cited by 28
Spatio-Temporal Feature Alignment Transfer Learning for Cross-Turbine Blade Icing Detection of Wind Turbines
Yue, R., Jiang, G., Jin, X., He, Q., & Xie, P. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3350147 · Cited by 27
Dysbiosis of the gut microbiota in glioblastoma patients and potential biomarkers for risk assessment
Jiang, H., Yang, F., Zhang, X., Fang, H., Qiu, T., Li, Y., & Peng, A. (2024) · Microbial Pathogenesis · doi:10.1016/j.micpath.2024.106888 · Cited by 25
A Systematic Review and Critical Assessment of Breast Cancer Risk Prediction Tools Incorporating a Polygenic Risk Score for the General Population
Mbuya-Bienge, C., Pashayan, N., Kazemali, C. D., Lapointe, J., Simard, J., & Nabi, H. (2023) · Cancers · doi:10.3390/cancers15225380 · Cited by 24
Analysis of the role of PANoptosis in seizures via integrated bioinformatics analysis and experimental validation
Liu, Y., Chang, Y., Jiang, X., Mei, H., Cao, Y., Wu, D., Xie, R., Jiang, W., Vasquez, E., Wu, Y., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26219 · Cited by 25
Comprehensive Assessment of Drought Susceptibility Using Predictive Modeling, Climate Change Projections, and Land Use Dynamics for Sustainable Management
Liu, J., Li, M., Li, R., Shalamzari, M. J., Ren, Y., & Silakhori, E. (2025) · Land · doi:10.3390/land14020337 · Cited by 24
Sensitivity of C-Reactive Protein and Procalcitonin Measured by Point-of-Care Tests to Diagnose Urinary Tract Infections in Nursing Home Residents: A Cross-Sectional Study
Kuil, S. D., Hidad, S., Fischer, J., Harting, J., Hertogh, C. M. P. M., Prins, J. M., de Jong, M. D., van Leth, F., & Schneeberger, C. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa1709 · Cited by 23
Prediction models for functional status in community dwelling older adults: a systematic review
Grootven, B. V., & van Achterberg, T. (2022) · BMC Geriatrics · doi:10.1186/s12877-022-03156-7 · Cited by 24
Discriminative performance of pancreatic stone protein in predicting ICU mortality and infection severity in adult patients with infection: a systematic review and individual patient level meta-analysis
Zuercher, P., Moser, A., de Guadiana‐Romualdo, L. G., Llewelyn, M., Graf, R., Reding, T., Eggimann, P., Que, Y., & Prazak, J. (2023) · Infection · doi:10.1007/s15010-023-02093-w · Cited by 24
External validation of the Memorial Sloan Kettering Cancer Centre and Briganti nomograms for the prediction of lymph node involvement of prostate cancer using clinical stage assessed by magnetic resonance imaging
Soeterik, T. F. W., Hueting, T. A., Israël, B., van Melick, H. H. E., Dijksman, L. M., Stomps, S. P., Biesma, D. H., Koffijberg, H., Sedelaar, J. P. M., Witjes, J. A., et al. (2021) · British Journal of Urology · doi:10.1111/bju.15376 · Cited by 21
The Misuse of AUC: What High Impact Risk Assessment Gets Wrong
Kwegyir-Aggrey, K., Gerchick, M., Mohan, M., Horowitz, A., & Venkatasubramanian, S. (2023) · Conference on Fairness, Accountability and Transparency · doi:10.1145/3593013.3594100 · Cited by 22
Machine Learning Models for Predicting Adverse Pregnancy Outcomes in Pregnant Women with Systemic Lupus Erythematosus
Hao, X., Zheng, D., Khan, M., Wang, L., Hämäläinen, T., Cong, F., Xu, H., & Song, K. (2023) · Diagnostics · doi:10.3390/diagnostics13040612 · Cited by 17
Six Externally Validated Prognostic Models Have Potential Clinical Value to Predict Patient Health Outcomes in the Rehabilitation of Musculoskeletal Conditions: A Systematic Review
Naye, F., Décary, S., Houle, C., LeBlanc, A., Cook, C. E., Dugas, M., Skidmore, B., & Tousignant‐Laflamme, Y. (2023) · Physical Therapy · doi:10.1093/ptj/pzad021 · Cited by 16
External validation of 87 clinical prediction models supporting clinical decisions for breast cancer patients
Hueting, T. A., van Maaren, M. C., Hendriks, M. P., Koffijberg, H., & Siesling, S. (2023) · The Breast · doi:10.1016/j.breast.2023.04.003 · Cited by 18
Artificial Intelligence in the Early Prediction of Cardiogenic Shock in Acute Heart Failure or Myocardial Infarction Patients: A Systematic Review and Meta-Analysis
Popat, A., Yadav, S., Patel, S. K., Baddevolu, S., Adusumilli, S., Dasari, N. R., Sundarasetty, M., Anand, S., Sankar, J., & Jagtap, Y. G. (2023) · Cureus · doi:10.7759/cureus.50395 · Cited by 15
Brachial-ankle pulse wave velocity is a stronger predictor than blood pressure for atherosclerotic cardiovascular diseases and all-cause mortality: a cohort study
Yang, Y., Li, D., Liu, R., Hu, Y., Chen, S., Wu, S., & Tian, Y. (2023) · Hypertension Research · doi:10.1038/s41440-023-01313-y · Cited by 16
A deep learning hybrid ensemble fusion for chest radiograph classification
Sultana, S., Hussain, S. S., Hashmani, M. A., Ahmad, J., & Zubair, M. (2021) · Neural Network World · doi:10.14311/nnw.2021.31.010 · Cited by 13
Spatial mapping Zataria multiflora using different machine-learning algorithms
Edalat, M., Dastres, E., Jahangiri, E., Moayedi, G., Zamani, A., Pourghasemi, H. R., & Tiefenbacher, J. P. (2022) · CATENA · doi:10.1016/j.catena.2021.106007 · Cited by 13
Predicting retail customers' distress in the finance industry: An early warning system approach
Beltman, J., Machado, M., & Osterrieder, J. R. (2024) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2024.104101 · Cited by 12
Can AI-based body composition assessment outperform body surface area in predicting dose-limiting toxicities for colonic cancer patients on chemotherapy?
Cao, K., Yeung, J., Arafat, Y., Choi, C., Wei, M. Y.-K., Chan, S. T. F., Lee, M., Baird, P. N., & Yeung, J. M. C. (2023) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-023-05227-7 · Cited by 12
Comparison and interpretability of machine learning models to predict severity of chest injury
Kulshrestha, S., Dligach, D., Joyce, C., Gonzalez, R., O’Rourke, A. P., Glazer, J. M., Stey, A. M., Kruser, J. M., Churpek, M. M., & Afshar, M. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab015 · Cited by 12
Deep Sparse Topology Network for Robust Bearing Fault Diagnosis by Maximizing Prior Knowledge Functions
Qian, W., Li, S., & Lu, J. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3148993 · Cited by 12
Development of machine learning models to predict cancer-related fatigue in Dutch breast cancer survivors up to 15 years after diagnosis
Beenhakker, L., Wijlens, K. A. E., Witteveen, A., Heins, M. J., Korevaar, J. C., de Ligt, K. M., Bode, C., Vollenbroek-Hutten, M. M. R., & Siesling, S. (2023) · Journal of Cancer Survivorship · doi:10.1007/s11764-023-01491-1 · Cited by 10
Stratified care in hip arthroscopy: can we predict successful and unsuccessful outcomes? Development and external temporal validation of multivariable prediction models
Ishøi, L., Thorborg, K., Kallemose, T., Kemp, J. L., Reiman, M. P., Nielsen, M. F., & Hölmich, P. (2023) · British Journal of Sports Medicine · doi:10.1136/bjsports-2022-105534 · Cited by 9
Habitat suitability modeling of dominant weed in canola (Brassica napus) fields using machine learning techniques
Dastres, E., Sarvestani, G. S., Edalat, M., & Pourghasemi, H. R. (2025) · Weed Science · doi:10.1017/wsc.2025.5 · Cited by 9
Risk analysis and risk prediction of in-hospital heart failure in patients with acute myocardial infarction after emergency intervention surgery
Song, H., Qin, W.-B., Yang, F., Tang, W.-Z., & He, G. (2024) · BMC Cardiovascular Disorders · doi:10.1186/s12872-024-04357-1 · Cited by 8
Developing Economic Models for Assessing the Cost-Effectiveness of Multiple Diagnostic Tests: Methods and Applications
Xie, X., Tiggelaar, S., Guo, J., Wang, M., Vandersluis, S., & Ungar, W. J. (2022) · Medical Decision Making · doi:10.1177/0272989x221089268 · Cited by 9
Limitations of Separating Athletes into High or Low-Risk Groups based on a Cut-Off. A Clinical Commentary
Losciale, J. M., Truong, L. K., Ward, P., Collins, P. G. S., & Bullock, G. S. (2024) · International Journal of Sports Physical Therapy · doi:10.26603/001c.122644 · Cited by 7
Modelling key ecological factors influencing the distribution and content of silymarin antioxidant in Silybum marianum L.
Hojati, M., Naderi, R., Edalat, M., & Pourghasemi, H. R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0322442 · Cited by 8
Identification of Rainfall Thresholds Likely to Trigger Flood Damages across a Mediterranean Region, Based on Insurance Data and Rainfall Observations
Papagiannaki, K., Kotroni, V., Lagouvardos, K., Bezes, A., Vafeiadis, V., Messini, I., Kroustallis, E., & Totos, I. (2022) · Water · doi:10.3390/w14060994 · Cited by 7
Platelets lipidomics study of blood stasis rat model by using liquid chromatography‐tandem mass spectrometry
Sun, M., Li, Y., Hu, G., Yang, H., Zhang, G., Miao, L., Zhang, Y., Li, L., & Liu, J. (2022) · Journal of Separation Science · doi:10.1002/jssc.202200611 · Cited by 7
Prediction begins with diagnosis: Estimating seizure recurrence risk in the First Seizure Clinic
Ooi, S., Tailby, C., Nagino, N., Carney, P. W., Jackson, G. D., & Vaughan, D. N. (2024) · Seizure · doi:10.1016/j.seizure.2024.09.013 · Cited by 6
ROC curves for clinical prediction models part 3. The ROC plot: a picture that needs a 1000 words
CALSTER, B. V., Wynants, L., Collins, P. G. S., Verbakel, J. Y. J., & Steyerberg, E. W. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.05.037 · Cited by 6
Self-citation
Metabolic score for visceral fat as a superior predictor of frailty in overweight and obese adults: a cross-sectional NHANES analysis (2007–2020)
Zhan, Y., Ripley-Gonzalez, J. W., Liu, Y., Gao, Y., Tang, J., & Gao, S. (2025) · Diabetology & Metabolic Syndrome · doi:10.1186/s13098-025-02016-9 · Cited by 5
Guidelines for setting cut-off scores in AUC (AUC-GUIDE): balancing sensitivity, specificity, and purpose
Bujang, M. A. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1808675 · Cited by 7
Establishment and validation of a nomogram model for riskprediction of hepatic encephalopathy: a retrospective analysis
Yao, C., Huang, L., Wang, M., Mao, D., Wang, M., Zheng, J., Long, F., Huang, J., Liu, X., Zhang, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-47012-z · Cited by 5
Aberrant functional connectivity patterns in the pregenual anterior cingulate cortex and anterior midcingulate cortex of patients with irritable bowel syndrome accompanied by depressive symptoms
Tang, R., Jin, Y., Xu, K., Wu, L., Chen, X., Guo, Y., Li, G., & Li, J. (2025) · Brain Imaging and Behavior · doi:10.1007/s11682-024-00964-w · Cited by 5
Pilot Study by Liquid Biopsy in Gastrointestinal Stromal Tumors: Analysis of PDGFRA D842V Mutation and Hypermethylation of SEPT9 Presence by Digital Droplet PCR
Olivera-Salazar, R., Cabañas, G. S., Vega-Clemente, L., Alonso-Martín, D., Megías, V. M. C., Volward, P., García‐Olmo, D., & García‐Arranz, M. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25126783 · Cited by 5
Differentiating COVID-19 and dengue from other febrile illnesses in co-epidemics: Development and internal validation of COVIDENGUE scores
Gérardin, P., Maillard, O., Bruneau, L., Accot, F., Legrand, F., Poubeau, P., MANAQUIN, R., Andry, F., Bertolotti, A., & Levin, C. (2021) · Travel Medicine and Infectious Disease · doi:10.1016/j.tmaid.2021.102232 · Cited by 5
Discovery of octahydropyrrolo [3,2‐b] pyridin derivative as a highly selective Type I inhibitor of FGFR3 over VEGFR2 by high‐throughput virtual screening
Wang, X., Ye, C., Li, E., Xu, L., Lin, W., & Chen, G. (2022) · Journal of Cellular Biochemistry · doi:10.1002/jcb.30357 · Cited by 5
In silico Exploration of Inhibition Mechanism of Lianhua Qingwen Formula (LQF) Interaction on SARS-CoV-2 Mpro
Xue, X., Wang, X., Ye, C., Gao, M., Peng, L., Yu, K., & Chen, G. (2024) · Chemical Research in Chinese Universities · doi:10.1007/s40242-024-4150-1 · Cited by 4
ADNEX-AI: automated extraction of ultrasound predictors for interpretable ovarian cancer risk stratification
Geysels, A., Garofalo, G., Timmerman, S., Ceusters, J., Buonomo, F., Fischerová, D., Testa, A. C., Moro, F., Sladkevicius, P., Jokubkiene, L., et al. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-01215-x · Cited by 4
Effects of CAD/CAM restorative materials and thickness on the depth of cure of various light-cured resin composite materials
Fasham, T., Attal, J., Gouze, H., Abdel-Gawad, S., Dursun, E., & François, P. (2024) · Dental Materials Journal · doi:10.4012/dmj.2024-053 · Cited by 4
Hybrid Sensing for Fatigue Detection Using Wearables and RF
Taylor, W., Hill, D., Adam, R., Cooper, J. M., Abbasi, Q. H., & Imran, M. A. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3404637 · Cited by 4
Stacked ensemble machine learning approach for electroencephalography based major depressive disorder classification using temporal statistics
Ahmed, N. N., Bhat, T. K., & Powar, O. S. (2024) · Systems Science & Control Engineering · doi:10.1080/21642583.2024.2427028 · Cited by 4
Detection of sFas, sCD137, and IL-27 Cytokines as Potential Biomarkers for Hepatocellular Carcinoma Diagnosis
Do, X.-H., Le, M. T., Nguyen, T. H., Le, T. T., Nguyen, X., Mai, T. B., Hoang, T. M. N., & Than, U. T. T. (2023) · Journal of Hepatocellular Carcinoma · doi:10.2147/jhc.s409649 · Cited by 4
Predictive nomograms for risk and prognostic factors in metastatic bladder cancer: a population-based study
Shi, S., Peng, G., Luo, L., & Li, D. (2023) · Translational Cancer Research · doi:10.21037/tcr-23-1229 · Cited by 4
Medical Relevance, State-of-the-Art and Perspectives of “Sweet Metacode” in Liquid Biopsy Approaches
Pinkeová, A., Košútová, N., Jáné, E., Lorencová, L., Bertóková, A., Bertók, T., & Tkáč, J. (2024) · Diagnostics · doi:10.3390/diagnostics14070713 · Cited by 4
Preoperatively predicting failure to achieve the minimum clinically important difference and substantial clinical benefit for total knee arthroplasty patients using machine learning
Park, J., Miley, E. N., Zhong, X., & Gray, C. F. (2025) · Knee Surgery and Related Research · doi:10.1186/s43019-025-00289-y · Cited by 3
Development, validation and clinical utility of prognostic models for patients with traumatic brain injury in an Indian setting using machine learning and traditional approaches: a study protocol
Kamal, V. K., Agrawal, D. K., Kumar, A., Radhakrishnan, R., Ponnaiah, M., Ramaiya, D., Thiruvalluvan, A., & Sivaram, A. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-096275 · Cited by 3
Literature‐informed ensemble machine learning for three‐year diabetic kidney disease risk prediction in type 2 diabetes: Development, validation, and deployment of the PSMMC NephraRisk model
Tourkmani, A. M., Alharbi, T., Al-Ghamdi, A., Youzghadli, I. M., Alosaimi, F. S., & Azzam, A. Y. (2025) · Diabetes Obesity and Metabolism · doi:10.1111/dom.70385 · Cited by 3
Risk factors for preoperative nasal carriage of methicillin-resistant bacteria in total hip and knee arthroplasty patients
Tateiwa, T., Ishida, T., Takahashi, Y., Shinozaki, T., Masaoka, T., Shishido, T., Nishida, J., & Yamamoto, K. (2021) · Journal of Orthopaedic Science · doi:10.1016/j.jos.2021.04.014 · Cited by 3
Reliability, Validity, Modification and Expansion of the Chinese Version of the Disease-Specific Anxiety Questionnaire for Chronic Obstructive Pulmonary Disease
Miao, X., Han, Y., Wu, Z., Jin, X., Niu, M., Zhao, Q., & Lü, X. (2024) · International Journal of COPD · doi:10.2147/copd.s455171 · Cited by 3
Presenting predictions and performance of probabilistic models for clinical decision support in trauma care
Alptekin, C., Wohlgemut, J., Perkins, Z., Marsh, W., Tai, N., & Yet, B. (2024) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2024.105702 · Cited by 2
Combining artificial neural networks and a marginal structural model to predict the progression from depression to Alzheimer's disease
Orimaye, S. O., & Schmidtke, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1362230 · Cited by 3
Investigating soil erosion vulnerable zones based on clustered geoinformatics approach: a case study of Tyume River Catchment, Eastern Cape, South Africa
Mbatyoti, S., Sumner, P. D., Kalumba, A. M., Owolabi, S. T., & Belle, J. A. (2024) · Cuaternario y geomorfología: Revista de la Sociedad Española de Geomorfología y Asociación Española para el Estudio del Cuaternario · doi:10.17735/cyg.v38i3-4.101016 · Cited by 3
Risk scores in anaesthesia: the future is hard to predict
Drayton, D. J., Ayres, M., Relton, S. D., Sperrin, M., & Hall, M. (2022) · BJA Open · doi:10.1016/j.bjao.2022.100027 · Cited by 3
Integrated CT-derived fractional flow reserve and perivascular fat attenuation index: a multimodal approach to predict in-stent restenosis
Zhao, W., Huang, L., Zhu, M., Li, X., Liu, X., Zhang, W., Shao, C., & Li, Y. (2025) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2025.1703044 · Cited by 2
Electronic health record-based prediction models for dementia detection: a systematic review of model performance and quality
Lu, A., Srikanth, V. K., Westworth, S. E., Baey, Y.-G., Moran, C., Beare, R. J., Siostrom, K., Andrew, N. E., & Collyer, T. A. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag048 · Cited by 2
Development and Validation of Risk Prediction Model for New-Onset Diabetes After Percutaneous Coronary Intervention (NODAP): A Study Protocol for a Retrospective, Multicenter Analysis
Li, Y., Cui, J., Liu, Y., Chen, K., Huang, L., & Liu, Y. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.748256 · Cited by 2
Early Risk Pregnancy Prediction Based on Machine Learning Built on Intelligent Application Using Primary Health Care Cohort Data
Miranda, E., Kumbangsila, M., Aryuni, M., Richard, R., Zakiyyah, A. Y., & Sano, A. V. D. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-0248-4_11 · Cited by 2
Developing a prediction model to identify people with severe mental illness without regular contact to their GP - a study based on data from the Danish national registers
Naesager, A. H. D., Damgaard, S. N., Rozing, M. P., Siersma, V., Møller, A., & Jensen, K. T. (2024) · BMC Psychiatry · doi:10.1186/s12888-024-05743-x · Cited by 2
Machine Learning-Based Gene Expression Analysis to Identify Prognostic Biomarkers in Upper Tract Urothelial Carcinoma
Padullés, B., López-Aladid, R., Ingelmo‐Torres, M., Roldán, F. L., Martínez, C., Juez, J., Izquierdo, L., Mengual, L., & Alcaraz, A. (2025) · Cancers · doi:10.3390/cancers17162619 · Cited by 2
ROC Analyses Based on Measuring Evidence Using the Relative Belief Ratio
Al‐Labadi, L., Evans, M. J., & Liang, Q. (2022) · Entropy · doi:10.3390/e24121710 · Cited by 2
Predicting Retail Customers' Distress: Early Warning Systems and Machine Learning ApplicationsPreprint
Beltman, J., Osterrieder, J. R., & Machado, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4730470 · Cited by 2
Development and validation of a clinical prediction model for prognostic factors in patients with primary pontine hemorrhage
Hu, A., Qin, H., Wu, S., Zhao, X., Li, Y. Q., Chen, F., & Liu, T. (2024) · Brazilian Journal of Medical and Biological Research · doi:10.1590/1414-431x2024e13359 · Cited by 1
Prediction of Range of Motion in Patients After Total Knee Arthroplasty by Shear Wave Elastography
Kim, M.-W., & Lee, D.-H. (2025) · Bioengineering · doi:10.3390/bioengineering12101009 · Cited by 1
Modeling habitat suitability for saffron (Crocus sativus L.) using AI-based approaches to support sustainable industrial cultivation
Sarvestani, G. S., Edalat, M., Dastres, E., Naderi, R., & Pourghasemi, H. R. (2025) · Industrial Crops and Products · doi:10.1016/j.indcrop.2025.121974 · Cited by 1
Secondary Prevention of AFAIS: Deploying Traditional Regression, Machine Learning, and Deep Learning Models to Validate and Update CHA2DS2-VASc for 90-Day Recurrence
Simon, J. E., Kraiński, Ł., Karlinski, M. A., & Niewada, M. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14207327 · Cited by 1
Data enhancement to improve CNN accuracy for early detection of colorectal cancer
Athiyah, U., Rustad, S., Soeleman, M. A., Muljono, M., Akrom, M., & Konoralma, R. A. (2025) · Array · doi:10.1016/j.array.2025.100581 · Cited by 1
Enhancing happiness and life satisfaction in university students: analysis with a machine learning approach
Long, Q., Wang, Y., Axelin, A., Zheng, F., Cao, Z., Li, X., & Guo, J. (2025) · BMC Psychology · doi:10.1186/s40359-025-03809-3 · Cited by 1
Clinical Factors Associated With Current Relevance in Allergic Contact Dermatitis: Development of Predictive Models Based on Data From the Spanish Contact Dermatitis Register ( REIDAC )
Pesqué, D., de Frutos, F. J. O., Navarro‐Triviño, F. J., Sanz‐Sánchez, T., Zaragoza‐Ninet, V., Guijarro, S. C., Miquel‐Miquel, J., Salvador, J. F. S., González‐Pérez, R., González, I. R., et al. (2026) · Contact Dermatitis · doi:10.1111/cod.70089 · Cited by 1
Depth-stratified oxygen stress index for quantifying vertical hypoxia and enabling precise management in stratified reservoirs
Arzhangi, A., Partani, S., & Mehrjerdi, L. F. (2026) · Environmental Advances · doi:10.1016/j.envadv.2026.100694 · Cited by 1
Loss function influence on hyperparameter optimization for observational healthcare prediction models
Vereijken, F., Reps, J., Rijnbeek, P. R., & Williams, R. D. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag075 · Cited by 1
Differentiated pathways of heatwave adaptation: Urban–Rural contrasts in the role of regional income level and energy consumption on adaptive behavior
Zhang, H., Shi, Q., Xiao, C., Wang, J., & (21935), N. W. (2026) · Habitat International · doi:10.1016/j.habitatint.2026.103865 · Cited by 1
Exploring the Impact of Hyperparameter Tuning on the Accuracy of Machine Learning-based Flood Susceptibility Mapping
Razavi-Termeh, S. V., Sadeghi-Niaraki, A., Pourzangbar, A., Kişi, Ö., Hussain, J., & Choi, S.-M. (2026) · Water Resources Management · doi:10.1007/s11269-026-04827-5 · Cited by 1
DATA MINING APPLICATION FOR FINANCIAL DECISION OPTIMIZATION AT RISK
KOÇOĞLU, E., & Ersöz, F. (2021) · International Journal of 3D Printing Technologies and Digital Industry · doi:10.46519/ij3dptdi.950062 · Cited by 1
The correlation between the inner canthal distance and maxillary mesiodens in children
Tadano, M., Matsunaga, Y., Saito, K., Suzuki, Y., Nakamura, T., Hoshikawa, S., Chiba, M., Hino, R., Maruya, Y., Fukumoto, E., et al. (2023) · Pediatric Dental Journal · doi:10.1016/j.pdj.2023.04.001 · Cited by 1
Construction and validation of a postoperative hypothermia prediction model in elderly patients undergoing colorectal surgery
Wang, Y., Yang, N. Y., Zhang, J., Han, W.-Y., & Li, X. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e32391 · Cited by 1
Human activity recognition based on machine learning and FMCW radar
Faraj, A. A., Al-Nakkash, A. H., & Wadday, A. G. (2024) · AIP conference proceedings · doi:10.1063/5.0236567 · Cited by 1
LCLN-CA: A Survival Regression Analysis-Based Prediction Method for Catechin Content in Yunnan Sun-Dried Tea
Li, H., Wang, Q., Wang, H., Li, L., Wang, X., Wu, T., Wang, C., Qian, Y., Wang, X., Xia, Y., et al. (2024) · Horticulturae · doi:10.3390/horticulturae10121321 · Cited by 1
PIWI-interacting RNA biomarkers in gastrointestinal disease
Huang, X.-Y., Chen, S., Wang, Z., Lu, Y.-S., Liu, C.-T., & Chen, S.-Z. (2025) · Clinica Chimica Acta · doi:10.1016/j.cca.2025.120182 · Cited by 1
Machine Learning Techniques in Bipolar Disorder
Rebouças, D. B., Barreto, P., Noronha, L. T., Roza, T. H., & Passos, I. C. (2025) · Springer eBooks · doi:10.1007/978-3-031-85519-1_40 · Cited by 2
Predicting mortality by comorbidity for patients with hip arthroplasty: Prospective observational register studies of a nationwide Swedish cohort
Bülow, E. (2020) · Gothenburg University Publications Electronic Archive (Gothenburg University) · Cited by 1
Bi-dimensional Health Space Mapping: Machine Learning Analysis of Health Dynamics in Korean and Dutch CohortsPreprint
Kim, Y. J., van den Broek, T., Brouwer-Brolsma, E. M., Feskens, E. J. M., Bouwman, J., & Kwon, O. (2025) · Research Square · doi:10.21203/rs.3.rs-6960301/v1 · Cited by 1
Developing, validating, and evaluating clinical prediction models in breast and prostate cancer
Hueting, T. A. (2022) · doi:10.3990/1.9789036553940
Predicting hospital readmission risk from Electronic Health Records data for patients with Multiple Chronic Conditions: a validation study (Preprint)Preprint
Rafiq, M., Mazzocato, P., Guttmann, C., Spaak, J., & Savage, C. (2022) · doi:10.2196/preprints.39847

Authors: Verbakel, J. Y., Steyerberg, E. W., Uno, H., De Cock, B., Wynants, L., Collins, G. S., & Van Calster, B.
Affiliations: University of Oxford (GB)
Leiden University Medical Center (NL)
Dana-Farber Cancer Institute (US)
KU Leuven (BE)
ORCID: Verbakel, Jan Y.
Steyerberg, Ewout W.
Uno, Hajime
Wynants, Laure
Venue: Journal of Clinical Epidemiology
Publisher: Elsevier BV
Published: 2020-10
Volume: 126
Pages: 207-216
ISSN: 0895-4356 (ISSN-L)
DOI: 10.1016/j.jclinepi.2020.01.028
OpenAlex: W3045258609
Cited by: 138
References: 48
Citations by year: 2026: 26 · 2025: 34 · 2024: 29 · 2023: 21 · 2022: 15 · 2021: 9 · 2020: 3
FWCI: 8.46
Type: journal article
Language: en
Update notices: erratum (2021-02-01)
Open access: hybrid
doi.org
Licence: cc-by-nc-nd
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
www.elsevier.com/legal/tdmrep-license (tdm)
doi.org/10.15223/policy-017 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
doi.org/10.15223/policy-012 (stm-asf)
doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-004 (stm-asf)
Locations: Journal of Clinical Epidemiology (publishedVersion, cc-by-nc-nd)
PubMed (publishedVersion)
Leiden Repository (Leiden University) (submittedVersion, cc-by-nc-nd)
Lirias (KU Leuven) (submittedVersion, cc-by-nc-nd)
Data Archiving and Networked Services (DANS) (submittedVersion, cc-by-nc-nd)
Funders: Cancer Research UK
Patient-Centered Outcomes Research Institute
Fonds Wetenschappelijk Onderzoek
NIHR Oxford Biomedical Research Centre
Seventh Framework Programme
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
Keywords: ROC curve, area under the curve, clinical prediction model, risk prediction model, sensitivity and specificity, risk thresholds, model discrimination, external validation, testicular cancer
MeSH terms: Clinical Decision Rules, Computer Simulation, Decision Making, Humans, Lymph Nodes, Male, Models, Theoretical, Predictive Value of Tests, Retroperitoneal Neoplasms, ROC Curve, Sensitivity and Specificity, Testicular Neoplasms, Models, Statistical, Risk Assessment, Area Under Curve

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks