The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
There is no such thing as a validated prediction model
Open accessAbstract
BACKGROUND: Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? MAIN BODY: We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models. CONCLUSION: Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“There are widespread concerns that the performance of these tools in early validation studies may not reflect their use in practice.19,20”
From Uptake of Generative AI Integrated With Electronic Health Records in US Hospitals (2025) · cites this paper as [19]
In the citing paper
“Second, heterogeneity in model performance should be expected based on differences in populations and measurement procedures between locations, settings, or time periods.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [91]
In the citing paper
“Miscalibration is often not just a problem of the model but a sign that we need to improve our understanding of the various contexts in which the model is validated and used.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [91]
In the citing paper
“One successful external validation, however, does not establish transportability to many other settings, while such a requirement might lead to the selective reporting of validation data.100”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [100]
In the citing paper
“Using data from 2001 to 2011, EuroSCORE was shown to consistently overestimate mortality and its calibration deteriorated over time.100”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [100]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650
Anderson, D., Rodger, M., Ginsberg, J., Kearon, C., Gent, M., Turpie, A., Bormanis, J., Weitz, J., Chamberlain, M., Bowie, D., et al. (2000) · Thrombosis and Haemostasis · doi:10.1055/s-0037-1613830 · Cited by 1,651
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910
Self-citationUnderstanding receiver operating characteristic (ROC) curves
Fan, J., Upadhye, S., & Worster, A. (2006) · Canadian Journal of Emergency Medicine · doi:10.1017/s1481803500013336 · Cited by 843
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665
Cited by
Sorted by citations
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
“76 To avoid ambiguity and harmonise terminology, we refer to validation as evaluation 74 in this article (box 1).”
MethodEvaluation of clinical prediction models (part 1): from development to external validation
Collins, G. S., Dhiman, P., Ma, J., Schlussel, M. M., Archer, L., Van Calster, B., Harrell, F. E., Martin, G. P., Moons, K. G. M., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-074819 · Cited by 798
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544
Evaluation of clinical prediction models (part 2): how to undertake an external validation study
Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., & Collins, P. G. S. (2024) · BMJ · doi:10.1136/bmj-2023-074820 · Cited by 565
(2024) · BMJ · doi:10.1136/bmj.q902 · Cited by 416
“76 To avoid ambiguity and harmonise terminology, we refer to validation as evaluation 74 in this article (box 1).”
Metadata
- Authors
- Van Calster, B., Steyerberg, E. W., Wynants, L., & van Smeden, M.
- Affiliations
- KU Leuven (BE)
- Maastricht University (NL)
- Venue
- BMC Medicine
- Published
- 2023-02-24
- Volume
- 21
- Issue
- 1
- Pages
- 70-70
- ISSN
- 1741-7015 (ISSN-L)
- PMC
- PMC9951847
- OpenAlex
- W4321782827
- Cited by
- 286
- References
- 56
- Citations by year
- 2026: 99 · 2025: 91 · 2024: 67 · 2023: 29
- FWCI
- 58.07
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- Fonds Wetenschappelijk Onderzoek (G097322N)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- clinical prediction model, model validation, external validation, internal validation, model updating, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
