Prediction models need appropriate internal, internal-external, and external validation
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Temporal validation with calibration assessment is increasingly recommended but not yet routine in diabetes prediction modeling.
“an approach increasingly recommended but not yet routine in diabetes prediction modeling [94]”
From Machine learning-driven risk prediction for post-hospitalization diabetes case management: Integrating clinical and social determinants of health (2026) · cites this paper as [94]
In the main text (page not provided), the paper describes temporal external validation and recommends validation approaches that can split data by calendar time. It also discusses assessing calibration in external validation. It does not address diabetes prediction modeling or establish that temporal validation with calibration assessment is not yet routine in that field. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Austin, P. C., & Steyerberg, E. W. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280214558972 · Cited by 516 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 18
- [1]Tugwell P Knottnerus JA Transferability/generalizability deserves more attention in ‘retest’ studies in Diagnosis and Prognosis J Clin Epidemiol 2015 3 68 3 235 6 25726521
- [2]Tugwell P Knottnerus JA Clinical prediction models are not being validated J Clin Epidemiol 2015 1 68 1 1 2 25483238
- [3]Siontis GC Tzoulaki I Castaldi PJ Ioannidis JP External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination J Clin Epidemiol 2015 1 68 1 25 34 25441703
- [4]Martens FK Kers JG Janssens AC External validation is only needed when prediction models are worth it J Clin Epidemiol 2015 2 3 25726454
- [5]Siontis GC Ioannidis JP Response to letter: more rigorous, not less, external validation is needed J Clin Epidemiol 2015 1 31 25724895
- [6]Justice AC Covinsky KE Berlin JA Assessing the generalizability of prognostic information Ann Intern Med 1999 3 16 130 6 515 24 10075620
- [7]Harrell FE Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis Springer NY 2001
- [8]Steyerberg EW Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating Springer NY 2009
- [9]Steyerberg EW Harrell FE Jr Borsboom GJ Eijkemans MJ Vergouwe Y Habbema JD Internal validation of predictive models: efficiency of some procedures for logistic regression analysis J Clin Epidemiol 2001 8 54 8 774 81 11470385
- [10]Steyerberg EW Bleeker SE Moll HA Grobbee DE Moons KG Internal and external validation of predictive models: a simulation study of bias and precision in small samples J Clin Epidemiol 2003 5 56 5 441 7 12812818
- [12]Royston P Parmar MK Sylvester R Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer Stat Med 2004 3 30 23 6 907 26 15027080
- [13]Steyerberg EW Mushkudiani N Perel P Butcher I Lu J McHugh GS Murray GD Marmarou A Roberts I Habbema JD Maas AI Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics PLoS Med 2008 8 5 5 8 e165 18684008 PMC2494563
- [14]Balmaña J Stockwell DH Steyerberg EW Stoffel EM Deffenbaugh AM Reid JE Ward B Scholl T Hendrickson B Tazelaar J Burbidge LA Syngal S Prediction of MLH1 and MSH2 mutations in Lynch syndrome JAMA 2006 9 27 296 12 1469 78 17003395
- [15]Legrand C Duchateau L Janssen P Ducrocq V Sylvester R Validation of prognostic indices using the frailty model Lifetime Data Anal 2009 3 15 1 59 78 18618249
- [16]Debray TP Vergouwe Y Koffijberg H Nieboer D Steyerberg EW Moons KG A new framework to enhance the interpretation of external validation studies of clinical prediction models J Clin Epidemiol 2015 3 68 3 279 89 25179855
- [17]Steyerberg EW Vergouwe Y Towards better clinical prediction models: seven steps for development and an ABCD for validation Eur Heart J 2014 8 1 35 29 1925 31 24898551 PMC4155437
- [18]Collins GS Reitsma JB Altman DG Moons KG Transparent reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement J Clin Epidemiol 2015 2 68 2 134 43 25579640
- [19]Moons KG Altman DG Reitsma JB Ioannidis JP Macaskill P Steyerberg EW Vickers AJ Ransohoff DF Collins GS Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration Ann Intern Med 2015 1 6 162 1 W1 73 25560730
Cited by
Sorted by citations
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Self-citationCalculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Applications of digital technology in COVID-19 pandemic planning and response
Whitelaw, S., Mamas, M. A., Topol, E., & Van Spall, H. G. C. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30142-4 · Cited by 992 · see citing works
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887 · see citing works
Evaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798 · see citing works
Self-citationDeveloping clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Cabitza, F., Campagner, A., Soares, F., García de Guadiana-Romualdo, L., Challa, F., Sulejmani, A., Seghezzi, M., & Carobene, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106288 · Cited by 284 · see citing works
There is no such thing as a validated prediction model
Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M. (2023) · BMC Medicine · doi:10.1186/s12916-023-02779-w · Cited by 286 · see citing works
Koutsouleris, N., Dwyer, D. B., Degenhardt, F., Maj, C., Urquijo-Castro, M. F., Sanfelici, R., Popovic, D., Oeztuerk, O., Haas, S. S., Weiske, J., et al. (2021) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2020.3604 · Cited by 275 · see citing works
Evaluating the impact of prediction models: lessons learned, challenges, and recommendations
Kappen, T. H., van Klei, W. A., van Wolfswinkel, L., Kalkman, C. J., Vergouwe, Y., & Moons, K. G. M. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0033-6 · Cited by 215 · see citing works
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198 · see citing works
The Science of Prognosis in Psychiatry: A Review
Fusar-Poli, P., Hijazi, Z., Stahl, D., & Steyerberg, E. W. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2530 · Cited by 197 · see citing works
Self-citationWilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185 · see citing works
A population-level digital histologic biomarker for enhanced prognosis of invasive breast cancer
Amgad, M., Hodge, J. M., Elsebaie, M. A. T., Bodelon, C., Puvanesarajah, S., Gutman, D. A., Siziopikou, K. P., Goldstein, J. A., Gaudet, M. M., Teras, L. R., et al. (2024) · Nature Medicine · doi:10.1038/s41591-023-02643-7 · Cited by 137 · see citing works
Zhou, Q., Chen, Z.-H., Cao, Y.-H., & Peng, S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00524-2 · Cited by 126 · see citing works
Andaur Navarro, C. L., Damen, J. A. A., van Smeden, M., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.015 · Cited by 121 · see citing works
Validation of prediction models in the presence of competing risks: a guide through modern methods
van Geloven, N., Giardiello, D., Bonneville, E. F., Teece, L., Ramspek, C. L., van Smeden, M., Snell, K. I. E., van Calster, B., Pohar-Perme, M., Riley, R. D., et al. (2022) · BMJ · doi:10.1136/bmj-2021-069249 · Cited by 106 · see citing works
Self-citationSanfelici, R., Dwyer, D. B., Antonucci, L. A., & Koutsouleris, N. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.02.009 · Cited by 82 · see citing works
De Hond, A., Raven, W., Schinkelshoek, L., Gaakeer, M., Ter Avest, E., Sir, O., Lameijer, H., Hessels, R. A., Reijnen, R., De Jonge, E., et al. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104496 · Cited by 66 · see citing works
Self-citationKamran, F., Tang, S., Otles, E., McEvoy, D. S., Saleh, S. N., Gong, J., Li, B. Y., Dutta, S., Liu, X., Medford, R. J., et al. (2022) · BMJ · doi:10.1136/bmj-2021-068576 · Cited by 60 · see citing works
Prediction of Liver Transplant Rejection With a Biologically Relevant Gene Expression Signature
Levitsky, J., Kandpal, M., Guo, K., Zhao, L., Kurian, S., Whisenant, T., & Abecassis, M. (2022) · Transplantation · doi:10.1097/tp.0000000000003895 · Cited by 36 · see citing works
Nigatu, Y. T., Liu, Y., & Wang, J. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0971-x · Cited by 18 · see citing works
Predictive models for musculoskeletal injury risk: why statistical approach makes all the difference
Rhon, D. I., Teyhen, D. S., Collins, G. S., & Bullock, G. S. (2022) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2022-001388 · Cited by 20 · see citing works
Instability of the AUROC of clinical prediction models
van Leeuwen, F. D., Steyerberg, E. W., van Klaveren, D., Wessler, B., Kent, D. M., & van Zwet, E. W. (2025) · Statistics in Medicine · doi:10.1002/sim.70011 · Cited by 19 · see citing works
Self-citationAjuwon, B. I., Richardson, A., Roper, K., & Lidbury, B. A. (2023) · Viruses · doi:10.3390/v15081735 · Cited by 6 · see citing works
Prognostic models for COVID-19 needed updating to warrant transportability over time and space
van Klaveren, D., Zanos, T. P., Nelson, J., Levy, T. J., Park, J. G., Retel Helmrich, I. R. A., Rietjens, J. A. C., Basile, M. J., Hajizadeh, N., Lingsma, H. F., et al. (2022) · BMC Medicine · doi:10.1186/s12916-022-02651-3 · Cited by 5 · see citing works
A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools
Patel, H., Crusco, S., Hansen, D., Morales, J., Patale, T., Chiuzan, C., Sands, M., Naidich, J., Myers, K., Davidson, K. W., et al. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10381-y · Cited by 4 · see citing works
Lee, S.-Y., Saleem, M., Land, A. M., DeLaney, E. W., Garretson, A. R., Patel, M., Hall, A. G., Westrick, S., Ovalle, F., Cherrington, A. L., et al. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106604
- Authors
- Steyerberg, E. W., & Harrell, F. E.
- Affiliations
- Erasmus MC (NL)
- Vanderbilt University (US)
- Corresponding author
- Steyerberg, Ewout W.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2016-01
- Volume
- 69
- Pages
- 245-247
- ISSN
- 0895-4356 (ISSN-L)
- PMC
- PMC5578404
- OpenAlex
- W1980283223
- Cited by
- 1,304
- References
- 1
- Citations by year
- 2026: 327 · 2025: 159 · 2024: 149 · 2023: 144 · 2022: 143 · 2021: 150 · 2020: 94 · 2019: 61 · 2018: 35 · 2017: 17 · 2016: 19 · 2015: 5 · 2012: 1
- FWCI
- 22.55
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- PubMed Central (acceptedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- National Institute of Neurological Disorders and Stroke (U01NS086294)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- prediction model, model validation, internal validation, external validation, diagnosis, prognosis, clinical epidemiology
- MeSH terms
- Forecasting, Models, Theoretical, Validation Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
