Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
Prediction models are increasingly used to complement clinical reasoning and decision making in modern medicine in general, and in the cardiovascular domain in particular. Developed models first and foremost need to provide accurate and (internally and externally) validated estimates of probabilities of specific health conditions or outcomes in targeted patients. The adoption of such models must guide physician's decision making and an individual's behaviour, and consequently improve individual outcomes and the cost-effectiveness of care. In a series of two articles we review the consecutive steps generally advocated for risk prediction model research. This first article focuses on the different aspects of model development studies, from design to reporting, how to estimate a model's predictive performance and the potential optimism in these estimates using internal validation techniques, and how to quantify the added or incremental value of new predictors or biomarkers (of whatever type) to existing predictors. Each step is illustrated with empirical examples from the cardiovascular field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is therefore essential that key details of how a prediction model was developed and validated be clearly reported to enable synthesis and critical appraisal of all relevant information (14,(33)(34)(35)(36).”
From Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [14]
In the citing paper
“The TRIPOD Statement covers studies that report solely development [12,15], both development and external validation, and solely external validation (with or without updating), of a prediction model [14] (Figure 1).”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [15]
In the citing paper
“The TRIPOD Statement covers studies that report solely development,12,15 both development and external validation, and solely external validation (with or without updating), of a prediction model14 (Figure 3).”
From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [15]
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Harrell, F. E. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,195
Research works printed but not linked: 2
- Wells, 1997, A simple clinical model to categorize pretest probability in patients with suspected pulmonary embolism, Blood, 90, 424
- Hosmer DW Lemeshow S . Applied Survival Analysis: Regression Modelling of Time to Event Data. New York: Wiley, 1999.
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationRisk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195
Self-citationPrediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935
“We differentiated between articles reporting on the development 17-19 or external validation 19-21 of one or more prediction models (box 1).”
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Surgery · doi:10.1002/bjs.9736 · Cited by 889
Self-citationExternal validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887
Collins, G. S., Dhiman, P., Andaur Navarro, C. L., Ma, J., Hooft, L., Reitsma, J. B., Logullo, P., Beam, A. L., Peng, L., Van Calster, B., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-048008 · Cited by 852
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Circulation · doi:10.1161/circulationaha.114.014508 · Cited by 793
Self-citationSample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K. G., de Groot, J. A., Collins, G. S., Altman, D. G., Eijkemans, M. J., & Reitsma, J. B. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 791
Self-citationHuang, Y., Liu, Z., He, L., Chen, X., Pan, D., Ma, Z., Liang, C., Tian, J., & Liang, C. (2016) · Radiology · doi:10.1148/radiol.2016152234 · Cited by 743
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729
“Key characteristics, such as calibration and discrimination, are widely recommended aspects to evaluate [9,12-15,26,45,46].”
BackgroundSelf-citationDebray, T. P. A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E. W., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.018 · Cited by 597
Self-citationRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585
Self-citationDeveloping prediction models for clinical use using logistic regression: an overview
Shipe, M. E., Deppen, S. A., Farjah, F., & Grogan, E. L. (2019) · Journal of Thoracic Disease · doi:10.21037/jtd.2019.01.25 · Cited by 512
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Urology · doi:10.1016/j.eururo.2014.11.025 · Cited by 439
Self-citationPavlou, M., Ambler, G., Seaman, S., De Iorio, M., & Omar, R. Z. (2016) · Statistics in Medicine · doi:10.1002/sim.6782 · Cited by 418
Mungroop, T. H., van Rijssen, L. B., van Klaveren, D., Smits, F. J., van Woerden, V., Linnemann, R. J., de Pastena, M., Klompmaker, S., Marchegiani, G., Ecker, B. L., et al. (2019) · Annals of Surgery · doi:10.1097/sla.0000000000002620 · Cited by 414
Risk Prediction in Patients With Heart Failure
Rahimi, K., Bennett, D., Conrad, N., Williams, T. M., Basu, J., Dwight, J., Woodward, M., Patel, A., McMurray, J., & MacMahon, S. (2014) · JACC: Heart Failure · doi:10.1016/j.jchf.2014.04.008 · Cited by 410
Self-citationPredicting hospital admission at emergency department triage using machine learning
Hong, W. S., Haimovich, A. D., & Taylor, R. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201016 · Cited by 326
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.010 · Cited by 324
Self-citationCowley, L. E., Farewell, D. M., Maguire, S., & Kemp, A. M. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 263
Than, M., Flaws, D., Sanders, S., Doust, J., Glasziou, P., Kline, J., Aldous, S., Troughton, R., Reid, C., Parsonage, W. A., et al. (2014) · Emergency Medicine Australasia · doi:10.1111/1742-6723.12164 · Cited by 253
Binuya, M. A. E., Engelhardt, E. G., Schats, W., Schmidt, M. K., & Steyerberg, E. W. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01801-8 · Cited by 246
Perkins, D. O., Jeffries, C. D., Addington, J., Bearden, C. E., Cadenhead, K. S., Cannon, T. D., Cornblatt, B. A., Mathalon, D. H., McGlashan, T. H., Seidman, L. J., et al. (2015) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu099 · Cited by 238
Fundamentals of Clinical Data Science
(2019) · Springer eBooks · doi:10.1007/978-3-319-99713-1 · Cited by 235
Diagnostic and prognostic prediction models
Hendriksen, J. M. T., Geersing, G. J., Moons, K. G. M., & de Groot, J. A. H. (2013) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.12262 · Cited by 229
Self-citationUnderstanding and checking the assumptions of linear regression: a primer for medical researchers
Casson, R. J., & Farmer, L. D. (2014) · Clinical & Experimental Ophthalmology · doi:10.1111/ceo.12358 · Cited by 226
Multinational development and validation of an early prediction model for delirium in ICU patients
Wassenaar, A., van den Boogaard, M., van Achterberg, T., Slooter, A. J. C., Kuiper, M. A., Hoogendoorn, M. E., Simons, K. S., Maseda, E., Pinto, N., Jones, C., et al. (2015) · Intensive Care Medicine · doi:10.1007/s00134-015-3777-2 · Cited by 219
Debray, T. P. A., Moons, K. G. M., Ahmed, I., Koffijberg, H., & Riley, R. D. (2013) · Statistics in Medicine · doi:10.1002/sim.5732 · Cited by 212
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13244 · Cited by 207
Self-citationExploration, Inference, and Prediction in Neuroscience and Biomedicine
Bzdok, D., & Ioannidis, J. P. A. (2019) · Trends in Neurosciences · doi:10.1016/j.tins.2019.02.001 · Cited by 204
Risk Models to Predict Chronic Kidney Disease and Its Progression: A Systematic Review
Echouffo-Tcheugui, J. B., & Kengne, A. P. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001344 · Cited by 200
Self-citationSystematic review of prediction models for delirium in the older adult inpatient
Lindroth, H., Bratzke, L., Purvis, S., Brown, R., Coburn, M., Mrkobrada, M., Chan, M. T. V., Davis, D. H. J., Pandharipande, P., Carlsson, C. M., et al. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019223 · Cited by 198
Bellou, V., Belbasis, L., Konstantinidis, A. K., Tzoulaki, I., & Evangelou, E. (2019) · BMJ · doi:10.1136/bmj.l5358 · Cited by 197
The Science of Prognosis in Psychiatry: A Review
Fusar-Poli, P., Hijazi, Z., Stahl, D., & Steyerberg, E. W. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2530 · Cited by 197
Mungroop, T. H., Klompmaker, S., Wellner, U. F., Steyerberg, E. W., Coratti, A., D’Hondt, M., de Pastena, M., Dokmak, S., Khatkov, I., Saint-Marc, O., et al. (2021) · Annals of Surgery · doi:10.1097/sla.0000000000003234 · Cited by 194
Prognostic models in obstetrics: available, but far from applicable
Kleinrouweler, C. E., Cheong-See, F. M., Collins, G. S., Kwee, A., Thangaratinam, S., Khan, K. S., Mol, B. W. J., Pajkrt, E., Moons, K. G. M., & Schuit, E. (2016) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2015.06.013 · Cited by 193
Self-citationJolani, S., Debray, T. P. A., Koffijberg, H., van Buuren, S., & Moons, K. G. M. (2015) · Statistics in Medicine · doi:10.1002/sim.6451 · Cited by 192
Self-citationWilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185
Jiang, Y., Chen, C., Xie, J., Wang, W., Zha, X., Lv, W., Chen, H., Hu, Y., Li, T., Yu, J., et al. (2018) · EBioMedicine · doi:10.1016/j.ebiom.2018.09.007 · Cited by 184
Risk Prediction Models for Mortality in Ambulatory Patients With Heart Failure: A Systematic Review
Alba, A. C., Agoritsas, T., Jankowski, M., Courvoisier, D., Walter, S. D., Guyatt, G. H., & Ross, H. J. (2013) · Circulation: Heart Failure · doi:10.1161/circheartfailure.112.000043 · Cited by 181
Machine learning for genetic prediction of psychiatric disorders: a systematic review
Bracher‐Smith, M., Crawford, K. L., & Escott‐Price, V. (2020) · Molecular Psychiatry · doi:10.1038/s41380-020-0825-2 · Cited by 177
Risk Models to Predict Hypertension: A Systematic Review
Echouffo-Tcheugui, J. B., Batty, G. D., Kivimäki, M., & Kengne, A. P. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067370 · Cited by 172
Self-citationLo, S. N., Ma, J., Scolyer, R. A., Haydu, L. E., Stretch, J. R., Saw, R. P. M., Nieweg, O. E., Shannon, K. F., Spillane, A. J., Ch’ng, S., et al. (2020) · Journal of Clinical Oncology · doi:10.1200/jco.19.02362 · Cited by 168
Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease
van Smeden, M., Heinze, G., CALSTER, B. V., Asselbergs, F. W., Vardas, P. E., Bruining, N., de Jaegere, P. P. T., Moore, J. H., Denaxas, S., Boulesteix, A., et al. (2022) · European Heart Journal · doi:10.1093/eurheartj/ehac238 · Cited by 165
Self-citationMothilal, R. K., Mahajan, D., Tan, C., & Sharma, A. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462597 · Cited by 164
Heus, P., Damen, J. A., Pajouheshnia, R., Scholten, R. J. P. M., Reitsma, J. B., Collins, G. S., Altman, D. G., Moons, K. G. M., & Hooft, L. (2018) · BMC Medicine · doi:10.1186/s12916-018-1099-2 · Cited by 163
Self-citationThe chemistry of the Maillard reaction under physiological conditions: a review.
Njoroge, F. G., & Monnier, V. M. (1989) · PubMed · Cited by 163
Detection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160
How to derive and validate clinical prediction models for use in intensive care medicine
Labarère, J., Renaud, B., & Fine, M. J. (2014) · Intensive Care Medicine · doi:10.1007/s00134-014-3227-6 · Cited by 160
Cardiovascular risk assessment: a global perspective
Zhao, D. H., Liu, J., Xie, W., & Qi, Y. (2015) · Nature Reviews Cardiology · doi:10.1038/nrcardio.2015.28 · Cited by 158
The rise of big clinical databases
Cook, J. A., & Collins, G. S. (2015) · British journal of surgery · doi:10.1002/bjs.9723 · Cited by 153
Criteria for the use of omics-based predictors in clinical trials: explanation and elaboration
McShane, L. M., Cavenagh, M. M., Lively, T., Eberhard, D. A., Bigbee, W. L., Williams, P. M., Mesirov, J. P., Polley, M. C., Kim, K. Y., Tricoli, J. V., et al. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-220 · Cited by 151
Cardiovascular Disease Risk Prediction in the HIV Outpatient Study
Thompson‐Paul, A. M., Lichtenstein, K. A., Armon, C., Palella, F. J., Skarbinski, J., Chmiel, J. S., Hart, R. L. D., Wei, S. C., Loustalot, F. V., Brooks, J. T., et al. (2016) · Clinical Infectious Diseases · doi:10.1093/cid/ciw615 · Cited by 151
Exhaled breath metabolomics as a noninvasive diagnostic tool for acute respiratory distress syndrome
Bos, L. D. J., Weda, H., Wang, Y., Knobel, H., Nijsen, T. M. E., Vink, T. J., Zwinderman, A. H., Sterk, P. J., & Schultz, M. J. (2014) · European Respiratory Journal · doi:10.1183/09031936.00005614 · Cited by 148
Left Ventricular Diastolic Dysfunction Predicts Mortality in Patients With Systemic Sclerosis
Tennøe, A. H., Murbræch, K., Andreassen, J. C., Fretheim, H., Garen, T., Gude, E., Andreassen, A., Aakhus, S., Molberg, Ø., & Hoffmann‐Vold, A. (2018) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.07.068 · Cited by 141
Life After Surviving Fontan Surgery: A Meta-Analysis of the Incidence and Predictors of Late Death
Poh, C. L., & d’Udekem, Y. (2017) · Heart Lung and Circulation · doi:10.1016/j.hlc.2017.11.007 · Cited by 137
Jeong, C. W., Jung, J.-W., Cha, W. H., Lee, B. K., Lee, S., Jeong, S. J., Hong, S. K., Byun, S., & Lee, S. E. (2013) · PLoS ONE · doi:10.1371/journal.pone.0065888 · Cited by 135
Chen, Y., Qian, X., Zhang, Y., Su, W., Huang, Y., Wang, X., Chen, X., Zhao, E., Han, L., & Ma, Y. (2022) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2022.840386 · Cited by 133
Sekiguchi, M., Oda, I., Taniguchi, H., Suzuki, H., Morita, S., Fukagawa, T., Sekine, S., Kushima, R., & Katai, H. (2016) · Journal of Gastroenterology · doi:10.1007/s00535-016-1180-6 · Cited by 131
Fazel, S., Chang, Z., Fanshawe, T. R., Långström, N., Lichtenstein, P., Larsson, H., & Mallett, S. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(16)00103-6 · Cited by 129
Prediction versus aetiology: common pitfalls and how to avoid them
van Diepen, M., Ramspek, C. L., Jager, K. J., Zoccali, C., & Dekker, F. W. (2017) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfw459 · Cited by 127
Prediction Modeling Methodology
Dankers, F. J. W. M., Traverso, A., Wee, L., & van Kuijk, S. M. J. (2018) · Springer eBooks · doi:10.1007/978-3-319-99713-1_8 · Cited by 127
A scoping review of artificial intelligence-based methods for diabetes risk prediction
Mohsen, F., Al-Absi, H. R. H., Yousri, N. A., El Hajj, N., & Shah, Z. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00933-5 · Cited by 125
van Rossum, P. S. N., Fried, D., Zhang, L., Hofstetter, W. L., van Vulpen, M., Meijer, G. J., Court, L. E., & Lin, S. H. (2016) · Journal of Nuclear Medicine · doi:10.2967/jnumed.115.163766 · Cited by 125
van Loon (2016) · doi:10.1097/md.0000000000003428 · Cited by 124
Developing and validating risk prediction models in an individual participant data meta-analysis
Ahmed, I., Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-3 · Cited by 124
Self-citationPrediction models for exacerbations in patients with COPD
Guerra, B., Gaveikaite, V., Bianchi, C., & Puhan, M. A. (2017) · European Respiratory Review · doi:10.1183/16000617.0061-2016 · Cited by 124
Collins, S. P., Jenkins, C. A., Harrell, F. E., Liu, D., Miller, K. F., Lindsell, C. J., Naftilan, A. J., McPherson, J., Maron, D. J., Sawyer, D. B., et al. (2015) · JACC Heart Failure · doi:10.1016/j.jchf.2015.05.007 · Cited by 124
How to conduct a systematic review and meta-analysis of prognostic model studies
Damen, J. A., Moons, K. G. M., van Smeden, M., & Hooft, L. (2022) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2022.07.019 · Cited by 123
Self-citationComparison between various fracture risk assessment tools
Leslie, W. D., & Lix, L. M. (2013) · Osteoporosis International · doi:10.1007/s00198-013-2409-3 · Cited by 113
Estimated dead space fraction and the ventilatory ratio are associated with mortality in early ARDS
Morales–Quinteros, L., Schultz, M. J., Bringué, J., Calfee, C. S., Camprubí–Rimblas, M., Cremer, O. L., Horn, J., van der Poll, T., Sinha, P., Artigas, A., et al. (2019) · Annals of Intensive Care · doi:10.1186/s13613-019-0601-0 · Cited by 113
Schreiber, K. L., Zinboonyahgoon, N., Xu, X., Spivey, T., King, T. A., Dominici, L. S., Partridge, A. H., Golshan, M. A., Strichartz, G. R., & Edwards, R. R. (2018) · Journal of Pain · doi:10.1016/j.jpain.2018.11.004 · Cited by 112
Himmelreich, J. C. L., Veelers, L., Lucassen, W. A. M., Schnabel, R. B., Rienstra, M., van Weert, H. C. P. M., & Harskamp, R. E. (2020) · EP Europace · doi:10.1093/europace/euaa005 · Cited by 110
Chalmers, R. L., Keay, L. J., Hickson-Curran, S. B., & Gleason, W. (2016) · Contact Lens and Anterior Eye · doi:10.1016/j.clae.2016.04.005 · Cited by 109
Bullock, G. S., Mylott, J. A., Hughes, T., Nicholson, K. F., Riley, R. D., & Collins, G. S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01698-9 · Cited by 108
Precision prognostics for the development of complications in diabetes
Schiborn, C., & Schulze, M. B. (2022) · Diabetologia · doi:10.1007/s00125-022-05731-4 · Cited by 108
Handling missing predictor values when validating and applying a prediction model to new patients
Hoogland, J., van Barreveld, M., Debray, T. P. A., Reitsma, J. B., Verstraelen, T. E., Dijkgraaf, M. G. W., & Zwinderman, A. H. (2020) · Statistics in Medicine · doi:10.1002/sim.8682 · Cited by 105
Pintye, J. C., Drake, A. L., Kinuthia, J., Unger, J. B., Matemo, D. N., Heffron, R. A., Barnabas, R. V., Kohler, P. K., McClelland, R. S., & John-Stewart, G. C. (2016) · Clinical Infectious Diseases · doi:10.1093/cid/ciw850 · Cited by 105
Koole, K., Brunen, D., van Kempen, P. M. W., Noorlag, R., de Bree, R., Lieftink, C., van Es, R. J. J., Bernards, R., & Willems, S. M. (2016) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-15-1874 · Cited by 104
Evaluating risk prediction models for adults with heart failure: A systematic literature review
Tanna, G. L. D., Wirtz, H. S., Burrows, K. L., & Globe, G. A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0224135 · Cited by 104
Artificial intelligence in clinical care amidst COVID-19 pandemic: A systematic review
Adamidi, E. S., Mitsis, K., & Nikita, K. S. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.010 · Cited by 104
Studerus, E., Ramyead, A., & Riecher‐Rössler, A. (2017) · Psychological Medicine · doi:10.1017/s0033291716003494 · Cited by 102
Introduction to Artificial Intelligence and Machine Learning for Pathology
Harrison, J. H., Gilbertson, J. R., Hanna, M. G., Olson, N. H., Seheult, J. N., Sorace, J. M., & Stram, M. N. (2021) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2020-0541-cp · Cited by 102
Cichosz, S. L., Johansen, M. D., & Hejlesen, O. (2016) · Journal of Diabetes Science and Technology · doi:10.1177/1932296815611680 · Cited by 101
Net Risk Reclassification P Values: Valid or Misleading?
PEPE, M. S., Janes, H., & Li, C. I. (2014) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/dju041 · Cited by 100
TRIPOD statement: a preliminary pre-post analysis of reporting and methods of prediction models
Najafabadi, A. H. Z., Ramspek, C. L., Dekker, F. W., Heus, P., Hooft, L., Moons, K. G. M., Peul, W. C., Collins, G. S., Steyerberg, E. W., & van Diepen, M. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-041537 · Cited by 99
Self-citationRisk Prediction Models for Lung Cancer: A Systematic Review
Gray, E. P., Teare, M. D., Stevens, J., & Archer, R. J. (2015) · Clinical Lung Cancer · doi:10.1016/j.cllc.2015.11.007 · Cited by 99
Risk prediction models for acute kidney injury following major noncardiac surgery: systematic review
Wilson, T., Quan, S., Cheema, K., Zarnke, K. B., Quinn, R., de Koning, L., Dixon, E., Pannu, N., & James, M. T. (2015) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfv415 · Cited by 99
Predicting postoperative pulmonary complications in the general population
Canet, J., & Gallart, L. (2013) · Current Opinion in Anaesthesiology · doi:10.1097/aco.0b013e32835e8acd · Cited by 98
Chun, M., Clarke, R., Cairns, B. J., Clifton, D. A., Bennett, D., Chen, Y., Guo, Y., Pei, P., Lv, J., Yu, C., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab068 · Cited by 98
Charvat, H., Sasazuki, S., Inoue, M., Iwasaki, M., Sawada, N., Shimazu, T., Yamaji, T., Tsugane, S., & for the JPHC Study Group (2015) · International Journal of Cancer · doi:10.1002/ijc.29705 · Cited by 97
Rate and predictors of progression in elite and viremic HIV-1 controllers
León, A., Pérez, I., Ruíz-Mateos, E., Benito, J. M., Leal, M., López‐Galíndez, C., Rallón, N. I., Alcamí, J., López-Aldeguer, J., Viciana, P., et al. (2016) · AIDS · doi:10.1097/qad.0000000000001050 · Cited by 95
Advanced neuroimaging and its role in predicting neurodevelopmental outcomes in very preterm infants
Parikh, N. A. (2016) · Seminars in Perinatology · doi:10.1053/j.semperi.2016.09.005 · Cited by 95
Metadata
- Authors
- Moons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E.
- Affiliations
- University Medical Center Utrecht (NL)
- The University of Sydney (AU)
- MRC Clinical Trials Unit at UCL (GB)
- University of Oxford (GB)
- Corresponding author
- Moons, Karel G M
- Venue
- Heart
- Publisher
- BMJ
- Published
- 2012-03-07
- Volume
- 98
- Issue
- 9
- Pages
- 683-690
- ISSN
- 1355-6037 (ISSN-L)
- 1468-201X
- OpenAlex
- W2146404132
- Cited by
- 889
- References
- 100
- Citations by year
- 2026: 42 · 2025: 52 · 2024: 59 · 2023: 59 · 2022: 92 · 2021: 87 · 2020: 75 · 2019: 46 · 2018: 76 · 2017: 53 · 2016: 58 · 2015: 68 · 2014: 56 · 2013: 44 · 2012: 18
- FWCI
- 9.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Heart (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Medical Research Council (MC_EX_G0800814)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- risk prediction model, model development, internal validation, optimism bias, cardiovascular risk prediction, predictive performance, clinical decision-making, cost-effectiveness
- MeSH terms
- Cardiovascular Diseases, Cost-Benefit Analysis, Decision Making, Decision Support Techniques, Humans, Models, Theoretical, Predictive Value of Tests, Risk, Reproducibility of Results, Biomarkers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.