The DOI matches its Crossref record; 54 of the 56 references it cites were found in a registry or an index.
Risk prediction models: II. External validation, model updating, and impact assessment
Abstract
Clinical prediction models are increasingly used to complement clinical reasoning and decision-making in modern medicine, in general, and in the cardiovascular domain, in particular. To these ends, developed models first and foremost need to provide accurate and (internally and externally) validated estimates of probabilities of specific health conditions or outcomes in the targeted individuals. Subsequently, the adoption of such models by professionals must guide their decision-making, and improve patient outcomes and the cost-effectiveness of care. In the first paper of this series of two companion papers, issues relating to prediction model development, their internal validation, and estimating the added value of a new (bio)marker to existing predictors were discussed. In this second paper, an overview is provided of the consecutive steps for the assessment of the model's predictive performance in new individuals (external validation studies), how to adjust or update existing models to local circumstances or with new predictors, and how to investigate the impact of the uptake of prediction models on clinical decision-making and patient outcomes (impact studies). Each step is illustrated with empirical examples from the cardiovascular field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The literature highly recommends external validation for all AIPM applications when applied to a new setting3,9,17,49,86.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [86]
In the citing paper
“It was recommended to carefully consider whether predictions should be presented in a directive fashion (by also proposing decisions), or in an assistive way (e.g., by only showing estimated probabilities)9,22,40,68,86,87.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [86]
In the citing paper
“Alternative designs are stepped-wedge trials9,19,86, before-after studies86, and observational studies5,19,56,68,86.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [86]
In the citing paper
“The literature identifies a randomized controlled trial (RCT) as the ideal comparative study design, randomizing patients individually or per cluster5,9,49,68,86.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [86]
In the citing paper
“Decision Analytical Modeling may give an initial estimate of clinical utility before commencing a full-blown impact study (see phases 2 and 3)68,86.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [86]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
Cardiovascular disease risk profiles
Anderson, K. M., Odell, P. M., Wilson, P. W. F., & Kannel, W. B. (1991) · American Heart Journal · doi:10.1016/0002-8703(91)90861-b · Cited by 2,163
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Self-citationDesign and analysis of stepped wedge cluster randomized trials
Hussey, M. A., & Hughes, J. P. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.05.007 · Cited by 1,393
Clinical Prediction Rules: Applications and Methodological Standards
Wasson, J. H., Sox, H. C., Neff, R. K., & Goldman, L. (1985) · New England Journal of Medicine · doi:10.1056/nejm198509263131306 · Cited by 1,357
Prognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337
Self-citationPrognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172
Self-citation
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
“Hence, if relevant, the systematic review should report the performance measures of the original and ‘‘rounded’’ models where this information is available in the published primary report [32].”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M.
- Affiliations
- University Medical Center Utrecht (NL)
- MRC Clinical Trials Unit at UCL (GB)
- University of Oxford (GB)
- The University of Sydney (AU)
- Corresponding author
- Moons, Karel G M
- Venue
- Heart
- Publisher
- BMJ
- Published
- 2012-03-07
- Volume
- 98
- Issue
- 9
- Pages
- 691-698
- ISSN
- 1355-6037 (ISSN-L)
- 1468-201X
- OpenAlex
- W2151056789
- Cited by
- 1,195
- References
- 56
- Citations by year
- 2026: 65 · 2025: 86 · 2024: 91 · 2023: 115 · 2022: 113 · 2021: 118 · 2020: 87 · 2019: 92 · 2018: 71 · 2017: 58 · 2016: 61 · 2015: 78 · 2014: 73 · 2013: 62 · 2012: 21
- FWCI
- 11.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Heart (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Medical Research Council (MC_EX_G0800814)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- risk prediction model, external validation, model updating, impact assessment, clinical decision-making, cardiovascular risk prediction, model adoption, cost-effectiveness, cardiovascular outcomes
- MeSH terms
- Cardiovascular Diseases, Decision Making, Delivery of Health Care, Humans, Models, Theoretical, Prognosis, Reproducibility of Results, Risk Assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Risk prediction models: II. External validation, model updating, and impact assessment. https://app.cochat.ai/papers/10.1136/heartjnl-2011-301247
- The paper
- Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012). Risk prediction models: II. External validation, model updating, and impact assessment. Heart. https://doi.org/10.1136/heartjnl-2011-301247
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.