The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
There is no such thing as a validated prediction model
Open accessAbstract
BACKGROUND: Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? MAIN BODY: We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models. CONCLUSION: Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“There are widespread concerns that the performance of these tools in early validation studies may not reflect their use in practice.19,20”
From Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals (2025) · cites this paper as [19]
In the citing paper
“Second, heterogeneity in model performance should be expected based on differences in populations and measurement procedures between locations, settings, or time periods.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [91]
In the citing paper
“Miscalibration is often not just a problem of the model but a sign that we need to improve our understanding of the various contexts in which the model is validated and used.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [91]
In the citing paper
“One successful external validation, however, does not establish transportability to many other settings, while such a requirement might lead to the selective reporting of validation data.100”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [100]
In the citing paper
“Using data from 2001 to 2011, EuroSCORE was shown to consistently overestimate mortality and its calibration deteriorated over time.100”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [100]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650
Anderson, D., Rodger, M., Ginsberg, J., Kearon, C., Gent, M., Turpie, A., Bormanis, J., Weitz, J., Chamberlain, M., Bowie, D., et al. (2000) · Thrombosis and Haemostasis · doi:10.1055/s-0037-1613830 · Cited by 1,651
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910
Self-citationUnderstanding receiver operating characteristic (ROC) curves
Fan, J., Upadhye, S., & Worster, A. (2006) · Canadian Journal of Emergency Medicine · doi:10.1017/s1481803500013336 · Cited by 843
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665
Riley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585
Steyerberg, E. W., Wiegers, E., Sewalt, C. A., Büki, A., Citerio, G., Keyser, V. D., Ercole, A., Kunzmann, K., Lanyon, L., Lecky, F. E., et al. (2019) · The Lancet Neurology · doi:10.1016/s1474-4422(19)30232-7 · Cited by 499
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467
Minimum sample size for external validation of a clinical prediction model with a binary outcome
Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9025 · Cited by 400
Agniel, D., Kohane, I. S., & Weber, G. M. (2018) · BMJ · doi:10.1136/bmj.k1479 · Cited by 391
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371
Collins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.010 · Cited by 336
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312
Deep learning predicts hip fracture using confounding patient and healthcare variables
Badgeley, M. A., Zech, J. R., Oakden-Rayner, L., Glicksberg, B. S., Liu, M., Gale, W., McConnell, M. V., Percha, B., Snyder, T. M., & Dudley, J. T. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0105-1 · Cited by 290
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246
Predictive analytics in health care: how can we know it works?
Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., & Collins, G. S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz130 · Cited by 223
Debray, T. P. A., Moons, K. G. M., Ahmed, I., Koffijberg, H., & Riley, R. D. (2013) · Statistics in Medicine · doi:10.1002/sim.5732 · Cited by 217
Systematic review of prediction models for delirium in the older adult inpatient
Lindroth, H. L., Bratzke, L. C., Purvis, S., Brown, R., Coburn, M., Mrkobrada, M., Chan, M. T. V., Davis, D. H. J., Pandharipande, P. P., Carlsson, C. M., et al. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019223 · Cited by 201
Minimum sample size for external validation of a clinical prediction model with a continuous outcome
Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021) · Statistics in Medicine · doi:10.1002/sim.8766 · Cited by 198
Detection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160
Hickey, G. L., Grant, S. W., Murphy, G. J., Bhabra, M., Pagano, D., McAllister, K., Buchan, I., & Bridgewater, B. (2013) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezs584 · Cited by 154
Luijken, K., Groenwold, R. H. H., CALSTER, B. V., Steyerberg, E. W., & van Smeden, M. (2019) · Statistics in Medicine · doi:10.1002/sim.8183 · Cited by 139
Informative missingness in electronic health record systems: the curse of knowing
Groenwold, R. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00077-0 · Cited by 139
Fesmire, F. M., Brown, M. D., Espinosa, J. A., Shih, R. D., Silvers, S. M., Wolf, S. J., & Decker, W. W. (2011) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2011.01.020 · Cited by 134
CALSTER, B. V., Valentin, L., Froyman, W., Landolfo, C., Ceusters, J., Testa, A. C., Wynants, L., Sladkevicius, P., Holsbeke, C. V., Domali, E., et al. (2020) · BMJ · doi:10.1136/bmj.m2614 · Cited by 133
Jenkins, D. A., Martin, G. P., Sperrin, M., Riley, R. D., Debray, T. P. A., Collins, G. S., & Peek, N. (2021) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00090-3 · Cited by 131
Min, J. H., Lee, M. W., Park, H. S., Lee, D. H., Park, H. J., Lim, S., Choi, S., Lee, J., Lee, J. E., Ha, S. Y., et al. (2020) · Radiology · doi:10.1148/radiol.2020201940 · Cited by 123
Cook, G., Royle, K., Pawlyn, C., Hockaday, A., Shah, V., Kaiser, M., Brown, S., Gregory, W. M., Child, J. A., Davies, F. E., et al. (2019) · The Lancet Haematology · doi:10.1016/s2352-3026(18)30220-5 · Cited by 113
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071018 · Cited by 101
Austin, P. C., Klaveren, D. V., Vergouwe, Y., Nieboer, D., Lee, D. S., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.05.007 · Cited by 98
Prognostic models will be victims of their own success, unless…
Lenert, M. C., Matheny, M. E., & Walsh, C. G. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz145 · Cited by 97
Nestor, B., McDermott, M. B. A., Boag, W., Berner, G., Naumann, T., Hughes, M. C., Goldenberg, A., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.00690 · Cited by 89
Takada, T., Nijman, S. W. J., Denaxas, S., Snell, K. I. E., Uijl, A., Nguyen, T., Asselbergs, F. W., & Debray, T. P. A. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.03.025 · Cited by 89
Steyerberg, E. W., Nieboer, D., Debray, T. P. A., & van Houwelingen, H. C. (2019) · Statistics in Medicine · doi:10.1002/sim.8296 · Cited by 77
de Jong, V. M. T., Rousset, R. Z., Antonio‐Villa, N. E., Buenen, A. G., CALSTER, B. V., Bello‐Chavolla, O. Y., Brunskill, N. J., Ćurčin, V., Damen, J. A., Fermín‐Martínez, C. A., et al. (2022) · BMJ · doi:10.1136/bmj-2021-069881 · Cited by 64
Debray, T. P. A., Collins, P. G. S., Riley, R. D., Snell, K. I. E., CALSTER, B. V., Reitsma, J. B., & Moons, K. G. M. (2023) · BMJ · doi:10.1136/bmj-2022-071058 · Cited by 64
Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., et al. (2022) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.008487 · Cited by 62
Stiell, I. G., Clement, C. M., O'Connor, A. M. C., Davies, B., Leclair, C., Sheehan, P. J., Clavet, T., Beland, C., MacKenzie, T., & Wells, G. A. (2010) · Canadian Medical Association Journal · doi:10.1503/cmaj.091430 · Cited by 53
Luijken, K., Wynants, L., van Smeden, M., CALSTER, B. V., Steyerberg, E. W., Groenwold, R. H. H., Timmerman, D., Bourne, T., & Ukaegbu, C. I. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.11.001 · Cited by 53
Austin, P. C., Klaveren, D. V., Vergouwe, Y., Nieboer, D., Lee, D. S., & Steyerberg, E. W. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0012-3 · Cited by 48
Wynants, L., Kent, D. M., Timmerman, D., Lundquist, C. M., & CALSTER, B. V. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0046-9 · Cited by 47
Clinical experience and pre-test probability scores in the diagnosis of pulmonary embolism
Iles, S. (2003) · QJM · doi:10.1093/qjmed/hcg027 · Cited by 46
The heterogeneity of concentrated prescribing behavior: Theory and evidence from antipsychotics
Berndt, E. R., Gibbons, R. S., Kolotilin, A., & Taub, A. L. (2014) · Journal of Health Economics · doi:10.1016/j.jhealeco.2014.11.003 · Cited by 45
Kim, D., Chung, J. W., Choi, J., Succi, M. D., Conklin, J., Longo, M. G. F., Ackman, J. B., Little, B. P., Petranović, M., Kalra, M. K., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-29437-8 · Cited by 45
Wynants, L., Vergouwe, Y., Huffel, S. V., Timmerman, D., & CALSTER, B. V. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280216668555 · Cited by 43
Measurement error in biomarkers: sources, assessment, and impact on studies.
White, E. (2011) · PubMed · Cited by 36
Verification of the harmonization of human epididymis protein 4 assays
Ferraro, S., Borille, S., Carnevale, A., Frusciante, E., Bassani, N., & Panteghini, M. (2016) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2015-1142 · Cited by 28
Screening for data clustering in multicenter studies: the residual intraclass correlation
Wynants, L., Timmerman, D., Bourne, T., Huffel, S. V., & CALSTER, B. V. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-128 · Cited by 27
Is your clinical prediction model past its sell by date?
Reynard, C., Jenkins, D. A., Martin, G. P., Kontopantelis, E., & Body, R. M. (2022) · Emergency Medicine Journal · doi:10.1136/emermed-2021-212224 · Cited by 7
Cited by
Sorted by citations
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
“76 To avoid ambiguity and harmonise terminology, we refer to validation as evaluation 74 in this article (box 1).”
MethodEvaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 565
(2024) · BMJ · doi:10.1136/bmj.q902 · Cited by 416
“76 To avoid ambiguity and harmonise terminology, we refer to validation as evaluation 74 in this article (box 1).”
Metadata
- Authors
- Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M.
- Affiliations
- KU Leuven (BE)
- Maastricht University (NL)
- Venue
- BMC Medicine
- Published
- 2023-02-24
- Volume
- 21
- Issue
- 1
- Pages
- 70-70
- ISSN
- 1741-7015 (ISSN-L)
- PMC
- PMC9951847
- OpenAlex
- W4321782827
- Cited by
- 286
- References
- 56
- Citations by year
- 2026: 99 · 2025: 91 · 2024: 67 · 2023: 29
- FWCI
- 58.07
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- Fonds Wetenschappelijk Onderzoek (G097322N)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- clinical prediction model, model validation, external validation, internal validation, model updating, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
