Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
Prediction models are increasingly used to complement clinical reasoning and decision making in modern medicine in general, and in the cardiovascular domain in particular. Developed models first and foremost need to provide accurate and (internally and externally) validated estimates of probabilities of specific health conditions or outcomes in targeted patients. The adoption of such models must guide physician's decision making and an individual's behaviour, and consequently improve individual outcomes and the cost-effectiveness of care. In a series of two articles we review the consecutive steps generally advocated for risk prediction model research. This first article focuses on the different aspects of model development studies, from design to reporting, how to estimate a model's predictive performance and the potential optimism in these estimates using internal validation techniques, and how to quantify the added or incremental value of new predictors or biomarkers (of whatever type) to existing predictors. Each step is illustrated with empirical examples from the cardiovascular field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is therefore essential that key details of how a prediction model was developed and validated be clearly reported to enable synthesis and critical appraisal of all relevant information (14,(33)(34)(35)(36).”
From Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [14]
In the citing paper
“The TRIPOD Statement covers studies that report solely development [12,15], both development and external validation, and solely external validation (with or without updating), of a prediction model [14] (Figure 1).”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [15]
In the citing paper
“The TRIPOD Statement covers studies that report solely development,12,15 both development and external validation, and solely external validation (with or without updating), of a prediction model14 (Figure 3).”
From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [15]
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Harrell, F. E. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,195
Research works printed but not linked: 2
- Wells, 1997, A simple clinical model to categorize pretest probability in patients with suspected pulmonary embolism, Blood, 90, 424
- Hosmer DW Lemeshow S . Applied Survival Analysis: Regression Modelling of Time to Event Data. New York: Wiley, 1999.
Cited by
Items 801–900 of 1,000
Prognostic markers for predicting outcomes
Hackshaw, A. K. (2014) · Wiley eBooks · doi:10.1002/9781118527122.ch9
Disability in activities of daily living: a multifactorial approach
den Ouden, M. E. M. (2013) · Utrecht University Repository (Utrecht University)
Opondo, C. (2014) · LSHTM Research Online (London School of Hygiene and Tropical Medicine) · doi:10.17037/pubs.02095789
Meta-analysis of clinical prediction models
Debray, T. P. A. (2013) · Utrecht University Repository (Utrecht University) · doi:10.7490/f1000research.445.1
Predicting the Risk of Lung Cancer in Never-smokers
Thompson, C. (2015) · Brock University Digital Repository (Brock University)
The Future of Prostate Cancer Risk Prediction
Muir, K. R., Lophatananon, A., Gnanapragasam, V. J., & Rees, J. (2015) · Current Epidemiology Reports · doi:10.1007/s40471-015-0056-2
Diagnosis of pulmonary injury and infection by exhaled breath analysis
Bos, L. D. J. (2014) · UvA-DARE (University of Amsterdam)
Secondary tests for stratification of risk for atherosclerosis
Wierzbicki, A. S., & Reynolds, T. (2013) · Current Medical Research and Opinion · doi:10.1185/03007995.2013.794133
Almanac 2013: stable coronary artery disease
Islam, S., & Timmis, A. (2013) · Wiener klinische Wochenschrift · doi:10.1007/s00508-013-0473-5
Diagnostic strategies in patients with lower abdominal complaints in primary care
Kok, L. (2014) · Utrecht University Repository (Utrecht University)
Markers to guide treatment decisions: Methods and applications in obstetrics and gynecology
Tajik, P. (2015) · UvA-DARE (University of Amsterdam)
Islam, S., & Timmis, A. (2014) · Revista Uruguaya de Cardiología
Henderson, R. A., & Timmis, A. (2011) · Revista Portuguesa de Cardiologia · doi:10.1016/j.repc.2011.09.012
Wick, J. (2015) · Libraries and Cultural Resources (University of Calgary) · doi:10.11575/prism/25653
Machado, G. (2013) · Americanae (AECID Library)
Associations Between Personal Cancer History and Lung Cancer Risk
Everett, K. D. (2015) · Brock University Digital Repository (Brock University)
Альманах-2013: стабільна ішемічна хвороба серця*
Islam, S., & Timmis, A. (2014) · Український кардіологічний журнал
Factors Associated with Small Aggressive Non-Small Cell Lung Carcinomas
Warkentin, M. T. (2015) · Brock University Digital Repository (Brock University)
Meta-analysis of risk prediction studies
Ahmed, I. (2015) · University of Birmingham Institutional Research Archive (University of Birmingham)
Competing Risks and Prognostic Stages in Cirrhosis
D’Amico, G. (2015) · Springer eBooks · doi:10.1007/978-3-319-23018-4_3
Risk stratification for women undergoing in-vitro fertilisation treatment
Dhillon‐Smith, R. K. (2016) · University of Birmingham Institutional Research Archive (University of Birmingham)
Mind the gap - surviving in the modern world
Wierzbicki, A. S. (2016) · International Journal of Clinical Practice · doi:10.1111/ijcp.12818
Risk Prediction of Acute Kidney Injury Following, Major, Non-Cardiac Surgery
Wilson, T. (2016) · Libraries and Cultural Resources (University of Calgary) · doi:10.11575/prism/27670
Bachlechner, U. (2016) · DepositOnce · doi:10.14279/depositonce-5537
Jensen, J. K., Khera, A. V., & Emdin, C. A. (2017) · Elsevier eBooks · doi:10.1016/b978-0-323-42880-4.00017-0
Li, G. (2016) · MacSphere (McMaster University) · doi:10.71548/11521
Farfan, A. M. J., Poma, P. V. U., & Albarracin, J. A. G. (2017) · renati
Risk prediction model: Statistical and artificial neural network approach
Paiman, N. A., Hariri, A., & Masood, I. (2017) · AIP conference proceedings · doi:10.1063/1.4981143
The diagnosis and classification of low back-related leg pain
Stynes, S. M. (2017) · Keele Research Repository (Keele University)
Ciniselli, C. M. (2017) · Archivio Istituzionale della Ricerca (Universita Degli Studi Di Milano) · doi:10.13130/c-m-ciniselli_phd2017-03-29
Refined Prognostication in Coronary Artery Disease Using Routine Laboratory Test Data
Gerling, M. C. (2016) · Libraries and Cultural Resources (University of Calgary) · doi:10.11575/prism/27571
Pintye, J. C. (2017) · ResearchWorks at the University of Washington (University of Washington)
Reply to: “Statistical and methodological issues”
Shin, T. M., Shaikh, W. R., Etzkorn, J. R., Sobanko, J. F., Margolis, D. J., Gelfand, J. M., Chu, E. Y., Elenitsas, R., & Miller, C. J. (2017) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2017.05.030
Addressing Uncertainties in Health Utilities
Chan, K. (2017) · TSpace (University of Toronto)
Nurses' contribution to prevent seclusion in acute mental health care – a prognostic study protocol
Doedens, P., & Maaskant, J. M. (2013) · Pure Amsterdam UMC · doi:10.13140/2.1.2835.1044
Development of Cardiovascular Risk Assessment Model for People with Type 2 Diabetes in Oman
Alrawahi, A. H. (2017) · Griffith Research Online · doi:10.25904/1912/3256
Kim, P. (2017) · Scholarship@Western (Western University)
Validating and Updating Lung Cancer Prediction Models
Gray, E. P. (2018) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York)
Yatsuya, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-801238-3.99611-9
Masconi, K. L. (2016) · SUNScholar (Stellenbosch University)
Armon, R. (2017) · Nottingham ePrints (University of Nottingham)
Prognostic radiological variables derived from oesophageal cancer staging investigations
Foley, K. (2017) · ORCA Online Research @Cardiff (Cardiff University)
ter Riet, G., & Puhan, M. A. (2018) · Barcelona Respiratory Network · doi:10.23866/brnrev:2017-0018
Predicting Recurrent Shoulder Instability After a First-time Traumatic Anterior Shoulder Dislocation
Olds, M. (2019) · Tuwhera (Auckland University of Technology)
Assessment of prognostic models for patients with acute pulmonary embolism
Élias, A. (2016) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-9gxwbgx05
Vorselaars, W. (2019) · Utrecht University Repository (Utrecht University)
Ruiz‐Ruigómez, M. (2019) · Institutional Repository of the University of Granada (University of Granada)
Optimization and Machine Learning Methods for Diagnostic Testing of Prostate Cancer
Merdan, S. (2018) · Deep Blue (University of Michigan)
Prehospital Prediction of Acute Coagulopathy of Trauma
Peltan, I. D. (2015) · ResearchWorks at the University of Washington (University of Washington)
Jiang, D., & Shen, Y. (2019) · Expert Review of Molecular Diagnostics · doi:10.1080/14737159.2019.1692656
Development of clinical prediction rules for reducing delays in the diagnosis of multiple myeloma
Koshiaris, C. (2018) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-ngmaob4ar
Papavarnavas, N. (2019) · Open University of Cape Town (University of Cape Town)
Incidence and prognosis of post-traumatic headache in adults
Cancelliere, C. (2017) · TSpace (University of Toronto)
Mahmud, M. (2020) · Brock University Digital Repository (Brock University)
Assessment of fracture risk tools in care home residents
Ihama, F. E. (2019) · Keele Research Repository (Keele University)
Implementing personalized obstetric care
van Montfort, P. (2020) · Research Publications (Maastricht University) · doi:10.26481/dis.20200219pm
Collins, G. S. (2014) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2014.07.018
Fathalla, K. M. (2019) · Aston Publications Explorer (Aston University) · doi:10.48780/publications.aston.ac.uk.00041178
Prostate cancer prognostication based on clinical and histopathological tumor features
Zelić, R. (2020) · Figshare
Learning from big imaging data to predict radiotherapy treatment outcomes and side-effects
Zhenwei, S. (2020) · Research Publications (Maastricht University) · doi:10.26481/dis.20200908zs
New developments in diagnosis and treatment of Small Renal Masses
Zondervan, P. J. (2020) · UvA-DARE (University of Amsterdam)
On the prediction and projection of cancer survival
Maringe, C. (2020) · LSHTM Research Online (London School of Hygiene and Tropical Medicine) · doi:10.17037/pubs.04657528
At-risk mental states for bipolar disorder
Farrow, J. E., Nery, F. G., Patino, L. R., & DelBello, M. P. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-819182-8.00020-x
Housseine, N. (2020) · Utrecht University Repository (Utrecht University) · doi:10.33540/263
Zeng, J., Cao, W., & Wang, L. (2021) · Journal of Shanghai Jiaotong University (Science) · doi:10.1007/s12204-021-2255-y
Kim, D., Chon, S., Choi, J. H., & Kwak, Y. H. (2021) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2021.03.007
Saya, S. (2021) · Minerva Access (University of Melbourne)
Moylett, S. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-822884-5.00008-8
Liang, Y., Chen, K.-H., Yang, J., Zhang, J., Peng, R.-R., Yan, C., Shen, D.-S., Qu, S., Li, L., & Zhu, X. (2020) · Research Square · doi:10.21203/rs.3.rs-36762/v1
Han, S. Y. S. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.65906
Risk Stratification for Adverse Events after Carotid Endarterectomy
Timmerman, N. (2021) · Utrecht University Repository (Utrecht University) · doi:10.33540/917
Pre-hospital Trauma Systems: Pushing the Boundaries of Precision Medicine
van der Sluijs, R. (2021) · Utrecht University Repository (Utrecht University) · doi:10.33540/511
Schmidt, A. F. (2014) · Utrecht University Repository (Utrecht University)
Prevention Trials: Challenges in Design, Analysis, and Interpretation of Prevention Trials
Jiang, S., & Colditz, G. A. (2021) · Springer eBooks · doi:10.1007/978-3-319-52677-5_96-1
Oral Squamous Cell Carcinoma of the Maxilla: On the way to patient-specific care
Slieker, F. J. B. (2022) · Utrecht University Repository (Utrecht University) · doi:10.33540/1174
Almanac 2013: stable coronary artery disease
Islam, S., & Timmis, A. (2013) · Seminars in Cardiovascular Medicine · doi:10.2478/semcard-2013-0009
Riley, R. D., Tierney, J. F., & Stewart, L. A. (2021) · Wiley eBooks · doi:10.1002/9781119333784.refs5
Singh, P., Adderley, N. J., Hazlehurst, J., Price, M. J., Tahrani, A. A., Nirantharakumar, K., & Bellary, S. (2021) · doi:10.2337/figshare.15173232
Endovascular Revascularization Failure in African American Patients: Developing a Predictive Model
Pierce, N. L., Rakel, B., Brown, G. D., Gilbertson‐White, S., Moon, C., & Segre, L. S. (2020) · doi:10.17077/etd.005490
Cheng, Y., Ma, X., Liu, X., Zhao, Y., Sun, Y., Zhang, D., Zhao, Q., Xu, Y.-K., & Zhou, Y. (2020) · Research Square · doi:10.21203/rs.3.rs-44228/v1
Prediction models, machine learning and missing data
Nijman, S. W. J. (2022) · Utrecht University Repository (Utrecht University) · doi:10.33540/1375
Di, J., Li, X., Yang, J., Li, L., & Yu, X. (2022) · Risk Management and Healthcare Policy · doi:10.2147/rmhp.s357606
Improving the recognition of sepsis in primary care
Loots, F. J. (2022) · Utrecht University Repository (Utrecht University) · doi:10.33540/1257
Aspirin for the prevention of spontaneous preterm birth
Landman, A. J. E. M. C. (2023) · doi:10.5463/thesis.10
Benats, F., Vaes, B., Hegendörfer, E., & Degryse, J. (2022) · Gériatrie et Psychologie Neuropsychiatrie du Viellissement · doi:10.1684/pnv.2022.1065
Head and neck cancer cachexia: A multidimensional approach
Willemsen, A. C. H. (2023) · Research Publications (Maastricht University) · doi:10.26481/dis.20230324aw
Identifying the critically ill pediatric oncology patient
Soeteman, M. (2024) · Utrecht University Repository (Utrecht University) · doi:10.33540/2063
Zakai, N. A., Wilkinson, K. S., Sparks, A. D., Gergi, M., Repp, A. B., Al-Samkari, H. T., Thomas, R., & Roetker, N. S. (2024) · Research and Practice in Thrombosis and Haemostasis · doi:10.1016/j.rpth.2024.102483
Prediction of Heart Problems in Diabetic Patients Using Machine Learning Algorithm
Ayyar, B. A. B., Bharanidharan, R., Ilakkia, N. J., Saravanakumar, K., Dhanaselvam, J., & Rajesh, R. (2024) · Algorithms for intelligent systems · doi:10.1007/978-981-97-3191-6_23
Rusli, R. A., Makmor‐Bakry, M., Shah, N. M., Loo, X. L., & Hung, S. K. Y. (2024) · Therapeutics and Clinical Risk Management · doi:10.2147/tcrm.s467975
van Hees, M. S. F., van Kuijk, S. M. J., Koppes, D. M., Oudijk, M. A., Vankan, E., Smits, L. J. M., & Scheepers, H. C. J. (2024) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2024.10.018
Geo-temporal Disease Visualization of Bangladesh from Empirical Data Using Machine Learning
Rushee, K. I., Hasan, K. T., Rozario, V. S., Nandi, D., & Fariha, F. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-3937-0_35
TENDÊNCIAS NA AVALIAÇÃO DO RACIOCÍNIO CLÍNICO: UMA ANÁLISE BIBLIOMÉTRICA ABRANGENDO 1974-2024
Ellwanger, M. P., Ellwanger, M. P., Tureck, F., Diogo, F. D. W., Oliveira, L., Silva, A. D. D. J., Reinert, D., Diogo, Á. V. D. S., & Munhoz, A. C. M. (2024) · Aracê. · doi:10.56238/arev6n2-187
Zhang, J., Flachskampf, F. A., Zhu, C.-Y., Chen, Y., Wu, M., Ren, Q. W., Huang, J., Guo, R., Gu, W., Hung, Y., et al. (2024) · European Heart Journal - Cardiovascular Imaging · doi:10.1093/ehjci/jeae281
Metadata
- Authors
- Moons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E.
- Affiliations
- University Medical Center Utrecht (NL)
- The University of Sydney (AU)
- MRC Clinical Trials Unit at UCL (GB)
- University of Oxford (GB)
- Corresponding author
- Moons, Karel G M
- Venue
- Heart
- Publisher
- BMJ
- Published
- 2012-03-07
- Volume
- 98
- Issue
- 9
- Pages
- 683-690
- ISSN
- 1355-6037 (ISSN-L)
- 1468-201X
- OpenAlex
- W2146404132
- Cited by
- 889
- References
- 100
- Citations by year
- 2026: 42 · 2025: 52 · 2024: 59 · 2023: 59 · 2022: 92 · 2021: 87 · 2020: 75 · 2019: 46 · 2018: 76 · 2017: 53 · 2016: 58 · 2015: 68 · 2014: 56 · 2013: 44 · 2012: 18
- FWCI
- 9.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Heart (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Medical Research Council (MC_EX_G0800814)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- risk prediction model, model development, internal validation, optimism bias, cardiovascular risk prediction, predictive performance, clinical decision-making, cost-effectiveness
- MeSH terms
- Cardiovascular Diseases, Cost-Benefit Analysis, Decision Making, Decision Support Techniques, Humans, Models, Theoretical, Predictive Value of Tests, Risk, Reproducibility of Results, Biomarkers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.