Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
Prediction models are increasingly used to complement clinical reasoning and decision making in modern medicine in general, and in the cardiovascular domain in particular. Developed models first and foremost need to provide accurate and (internally and externally) validated estimates of probabilities of specific health conditions or outcomes in targeted patients. The adoption of such models must guide physician's decision making and an individual's behaviour, and consequently improve individual outcomes and the cost-effectiveness of care. In a series of two articles we review the consecutive steps generally advocated for risk prediction model research. This first article focuses on the different aspects of model development studies, from design to reporting, how to estimate a model's predictive performance and the potential optimism in these estimates using internal validation techniques, and how to quantify the added or incremental value of new predictors or biomarkers (of whatever type) to existing predictors. Each step is illustrated with empirical examples from the cardiovascular field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is therefore essential that key details of how a prediction model was developed and validated be clearly reported to enable synthesis and critical appraisal of all relevant information (14,(33)(34)(35)(36).”
From Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [14]
In the citing paper
“The TRIPOD Statement covers studies that report solely development [12,15], both development and external validation, and solely external validation (with or without updating), of a prediction model [14] (Figure 1).”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [15]
In the citing paper
“The TRIPOD Statement covers studies that report solely development,12,15 both development and external validation, and solely external validation (with or without updating), of a prediction model14 (Figure 3).”
From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [15]
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Harrell, F. E. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,195
Research works printed but not linked: 2
- Wells, 1997, A simple clinical model to categorize pretest probability in patients with suspected pulmonary embolism, Blood, 90, 424
- Hosmer DW Lemeshow S . Applied Survival Analysis: Regression Modelling of Time to Event Data. New York: Wiley, 1999.
Cited by
Items 101–200 of 1,000
Exhaled Biomarkers and Gene Expression at Preschool Age Improve Asthma Prediction at 6 Years of Age
Klaassen, E. M. M., van de Kant, K. D. G., Jöbsis, Q., van Schayck, O. C. P., Smolinska, A., Dallinga, J. W., van Schooten, F., den Hartog, G. J. M., de Jongste, J. C., Rijkers, G. T., et al. (2015) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201408-1537oc · Cited by 95
Gerry, S., Birks, J., Bonnici, T., Watkinson, P., Kirtley, S., & Collins, G. S. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-019268 · Cited by 93
Herc, E. S., Patel, P., Washer, L., Conlon, A., Flanders, S. A., & Chopra, V. (2017) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2017.167 · Cited by 95
LLPi: Liverpool Lung Project Risk Prediction Model for Lung Cancer Incidence
Marcus, M. W., Chen, Y., Raji, O. Y., Duffy, S. W., & Field, J. K. (2015) · Cancer Prevention Research · doi:10.1158/1940-6207.capr-14-0438 · Cited by 89
Prognostic value of serum neurofilaments in patients with clinically isolated syndromes
Costa, G. D., Martinelli, V., Sangalli, F., Moiola, L., Colombo, B., Radaelli, M., Leocani, L., Furlan, R., & Comı, G. (2019) · Neurology · doi:10.1212/wnl.0000000000006902 · Cited by 89
Leijdekkers, J. A., Eijkemans, M. J. C., van Tilborg, T. C., Oudshoorn, S. C., McLernon, D. J., Bhattacharya, S., Mol, B. W., Broekmans, F. J., Torrance, H. L., & OPTIMIST group (2018) · Human Reproduction · doi:10.1093/humrep/dey263 · Cited by 88
Systematic review of predictive risk models for adverse drug events in hospitalized patients
Falconer, N., Barras, M. A., & Cottrell, N. (2018) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.13514 · Cited by 88
Delirium prediction in the intensive care unit: comparison of two delirium prediction models
Wassenaar, A., Schoonhoven, L., Devlin, J. W., van Haren, F. M. P., Slooter, A. J. C., Jorens, P. G., van der Jagt, M., Simons, K. S., Egerod, I., Burry, L. D., et al. (2018) · Critical Care · doi:10.1186/s13054-018-2037-6 · Cited by 89
Simplified prognostic model for critically ill patients in resource limited settings in South Asia
Haniffa, R., Mukaka, M., Munasinghe, S. B., Silva, A. P. D., Jayasinghe, K. S. A., Beane, A., de Keizer, N. F., & Dondorp, A. M. (2017) · Critical Care · doi:10.1186/s13054-017-1843-6 · Cited by 86
Carluccio, E., Dini, F. L., Biagioli, P., Lauciello, R., Simioniuc, A., Zuchi, C., Alunni, G., Reboldi, G., Marzilli, M., & Ambrosio, G. (2013) · European Journal of Heart Failure · doi:10.1093/eurjhf/hft038 · Cited by 86
Bours, M. J. L., Linden, B. W. A., Winkels, R. M., Duijnhoven, F. J., Mols, F., Roekel, E. H., Kampman, E., Beijer, S., & Weijenberg, M. P. (2016) · The Oncologist · doi:10.1634/theoncologist.2015-0258 · Cited by 85
Sahle, B. W., Owen, A. J., Chin, K. L., & Reid, C. M. (2017) · Journal of Cardiac Failure · doi:10.1016/j.cardfail.2017.03.005 · Cited by 85
Peters, K. E., Davis, W. A., Ito, J., Winfield, K., Stoll, T. M., Bringans, S., LIPSCOMBE, R. J., & Davis, T. M. E. (2017) · Diabetes Care · doi:10.2337/dc17-0911 · Cited by 85
Prediction models in in vitro fertilization; where are we? A mini review
van Loendersloot, L. L., Repping, S., Bossuyt, P. M., van der Veen, F., & van Wely, M. (2013) · Journal of Advanced Research · doi:10.1016/j.jare.2013.05.002 · Cited by 84
The potential of novel biomarkers to improve risk prediction of type 2 diabetes
Herder, C., Kowall, B., Tabák, Á. G., & Rathmann, W. (2013) · Diabetologia · doi:10.1007/s00125-013-3061-3 · Cited by 82
Hay, A. D., Redmond, N. M., Turnbull, S., Christensen, H., Thornton, H., Little, P. S., Thompson, M., Delaney, B. C., Lovering, A. M., Muir, P., et al. (2016) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(16)30223-5 · Cited by 82
A Systematic Review of Studies Comparing Diagnostic Clinical Prediction Rules with Clinical Judgment
Sanders, S., Doust, J. A., & Glasziou, P. (2015) · PLoS ONE · doi:10.1371/journal.pone.0128233 · Cited by 82
Pragmatism Instead of Mechanism: A Call for Impactful Biological Psychiatry
Paulus, M. P. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.0497 · Cited by 79
Wood, A. M., Royston, P., & White, I. R. (2015) · Biometrical Journal · doi:10.1002/bimj.201400004 · Cited by 78
“We focus on model performance within the derivation data, but describe how to extend all the methods to a validation dataset.”
MethodInfluentialSelf-citationden Ouden, M. E. M., Schuurmans, M. J., Mueller‐Schotte, S., & van der Schouw, Y. T. (2013) · Experimental Gerontology · doi:10.1016/j.exger.2013.02.002 · Cited by 78
Mao, S., Yu, X., Shan, Y., Fan, R., Wu, S., & Lu, C. (2021) · Journal of Hepatocellular Carcinoma · doi:10.2147/jhc.s339707 · Cited by 76
Pretransplant Risk Score for New-Onset Diabetes After Kidney Transplantation
Chakkera, H. A., Weil, E. J., Swanson, C. M., Dueck, A. C., Heilman, R. L., Reddy, K. S., Hamawi, K., Khamash, H. A., Moss, A. A., Mulligan, D. C., et al. (2011) · Diabetes Care · doi:10.2337/dc11-0752 · Cited by 76
Walsh, C., & Hripcsak, G. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.08.006 · Cited by 75
New horizons in arginine metabolism, ageing and chronic disease states
Mangoni, A. A., Rodionov, R. N., McEvoy, M., Zinellu, A., Carru, C., & Sotgia, S. (2019) · Age and Ageing · doi:10.1093/ageing/afz083 · Cited by 75
Su, H. I., Haunschild, C., Chung, K., Komrokian, S., Boles, S. G., Sammel, M. D., & DeMichele, A. M. (2014) · Cancer · doi:10.1002/cncr.28942 · Cited by 75
Predicting falls in community-dwelling older adults: a systematic review of prognostic models
Gade, G. V., Jørgensen, M. G., Ryg, J., Riis, J., Thomsen, K., Masud, T., & Andersen, S. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044170 · Cited by 74
Foley, K., Hills, R. K., Berthon, B., Marshall, C., Parkinson, C., Lewis, W. G., Crosby, T. D. L., Spezi, E., & Roberts, S. A. (2017) · European Radiology · doi:10.1007/s00330-017-4973-y · Cited by 74
Meta‐analysis and aggregation of multiple published prediction models
Debray, T. P. A., Koffijberg, H., Nieboer, D., Vergouwe, Y., Steyerberg, E. W., & Moons, K. G. M. (2014) · Statistics in Medicine · doi:10.1002/sim.6080 · Cited by 73
Self-citationAssessing Cardiovascular Risk in People Living with HIV: Current Tools and Limitations
Achhra, A. C., Lyass, A., Borowsky, L. H., Bogorodskaya, M., Plutzky, J., Massaro, J. M., D’Agostino, R. B., & Triant, V. A. (2021) · Current HIV/AIDS Reports · doi:10.1007/s11904-021-00567-w · Cited by 73
van Adrichem, R. A., Debeij, J., Nelissen, R. G. H. H., Schipper, I. B., Rosendaal, F. R., & Cannegieter, S. C. (2014) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.12655 · Cited by 72
Kidney Histopathology and Prediction of Kidney Failure: A Retrospective Cohort Study
Eadon, M. T., Schwantes-An, T.-H. L., Phillips, C. L., Roberts, A., Greene, C. V., Hallab, A., Hart, K., Lipp, S. N., Perez-Ledezma, C., Omar, K. O., et al. (2020) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2019.12.014 · Cited by 71
Riva, N., Ageno, W., Schulman, S., Beyer‐Westendorf, J., Duce, R., Malato, A., Santoro, R., Poli, D., Verhamme, P. B., Martinelli, I., et al. (2016) · The Lancet Haematology · doi:10.1016/s2352-3026(16)30020-5 · Cited by 71
Yeboah, J., Erbel, R., Delaney, J. C., Nance, R., Guo, M., Bertoni, A. G., Budoff, M., Moebus, S., Jöckel, K.-H., Burke, G. L., et al. (2014) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2014.07.035 · Cited by 70
Prediction models for depression risk among older adults: systematic review and critical appraisal
Tan, J., Ma, C., Zhu, C., Wang, Y., Zou, X., Li, H., Li, J., He, Y., & Wu, C. (2022) · Ageing Research Reviews · doi:10.1016/j.arr.2022.101803 · Cited by 71
Yang, H., Kim, S. G., Lim, J. H., Choi, J., Im, J. P., Kim, J. S., Kim, W. H., & Jung, H. C. (2014) · Surgical Endoscopy · doi:10.1007/s00464-014-3780-7 · Cited by 69
Nombo, A. P., Mwanri, A. W., Brouwer-Brolsma, E. M., Ramaiya, K. L., & Feskens, E. J. M. (2018) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2018.05.001 · Cited by 68
van Royen, F. S.-A., Moons, K. G. M., Geersing, G., & van Smeden, M. (2022) · European Respiratory Journal · doi:10.1183/13993003.00250-2022 · Cited by 68
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Journal of Clinical Investigation · doi:10.1111/eci.12376 · Cited by 58
Self-citationRisk Prediction Tools for Hip and Knee Arthroplasty
Manning, D. W., Edelstein, A. I., & Alvi, H. M. (2016) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-15-00072 · Cited by 66
Development of Clinical Prediction Models for Surgery and Complications in Crohn’s Disease
Guizzetti, L., Zou, G., Khanna, R., Dulai, P. S., Sandborn, W. J., Jairath, V., & Feagan, B. G. (2017) · Journal of Crohn s and Colitis · doi:10.1093/ecco-jcc/jjx130 · Cited by 67
Prediction Rules for Ruling Out Endocarditis in Patients With Staphylococcus aureus Bacteremia
van der Vaart, T. W., Prins, J. M., Soetekouw, R., Twillert, G. V., Veenstra, J., Herpers, B. L., Rozemeijer, W., Jansen, R. R., Bonten, M. J. M., & van der Meer, J. T. M. (2021) · Clinical Infectious Diseases · doi:10.1093/cid/ciab632 · Cited by 66
Wieske, L., Witteveen, E., Verhamme, C., Dettling-Ihnenfeldt, D. S., van der Schaaf, M., Schultz, M. J., van Schaik, I. N., & Horn, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0111259 · Cited by 63
The Importance of Calibration in Clinical Psychology
Lindhiem, O. J., Petersen, I. T., Mentch, L., & Youngstrom, E. A. (2018) · Assessment · doi:10.1177/1073191117752055 · Cited by 63
Goerdten, J., Čukić, I., Danso, S., Carrière, I., & Terrera, G. M. (2019) · Alzheimer s & Dementia Translational Research & Clinical Interventions · doi:10.1016/j.trci.2019.08.001 · Cited by 63
Park, H. J., Park, B., Park, S. Y., Choi, S. H., Rhee, H., Park, J. H., Cho, E., Yeom, S. K., Park, S., Park, M., et al. (2021) · European Radiology · doi:10.1007/s00330-021-07926-6 · Cited by 62
Lu, J. H., Callahan, A., Patel, B. S., Morse, K. E., Dash, D., Pfeffer, M. A., & Shah, N. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.27779 · Cited by 52
Palumbo, P., Klenk, J., Cattelani, L., Bandinelli, S., Ferrucci, L., Rapp, K., Chiari, L., & Rothenbacher, D. (2016) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2016.07.015 · Cited by 61
Ramspek, C. L., Voskamp, P. W. M., van Ittersum, F. J., Krediet, R. T., Dekker, F. W., & van Diepen, M. (2017) · Clinical Epidemiology · doi:10.2147/clep.s139748 · Cited by 64
Survival analyses in cardiovascular research, part I: the essentials
Rosselló, X., & González‐Del‐Hoyo, M. (2021) · Revista Española de Cardiología (English Edition) · doi:10.1016/j.rec.2021.06.003 · Cited by 61
Prediction modelling in the early detection of neonatal sepsis
Sahu, P., Stanly, E. A. R., Lewis, L., Prabhu, K., Rao, M., & Kunhikatta, V. (2022) · World Journal of Pediatrics · doi:10.1007/s12519-021-00505-1 · Cited by 62
Application of machine learning in predicting the risk of postpartum depression: A systematic review
Zhong, M., Zhang, H., Yu, C., Jiang, J., & Duan, X. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.08.070 · Cited by 59
Ball, T. M., Stein, M. B., & Paulus, M. P. (2014) · Depression and Anxiety · doi:10.1002/da.22299 · Cited by 53
Anaemia, Haemoglobin Level and Cause-Specific Mortality in People with and without Diabetes
Kengne, A. P., Czernichow, S., Hamer, M., Batty, G. D., & Stamatakis, E. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041875 · Cited by 59
Self-citationSeverity of thought disorder predicts psychosis in persons at clinical high-risk
Perkins, D. O., Jeffries, C. D., Cornblatt, B. A., Woods, S. W., Addington, J., Bearden, C. E., Cadenhead, K. S., Cannon, T. D., Heinssen, R., Mathalon, D. H., et al. (2015) · Schizophrenia Research · doi:10.1016/j.schres.2015.09.008 · Cited by 58
Wong, S. H., Petrie, A., & Plant, G. T. (2016) · Journal of Neuro-Ophthalmology · doi:10.1097/wno.0000000000000350 · Cited by 59
Análisis de supervivencia en investigación cardiovascular (I): lo esencial
Rosselló, X., & González‐Del‐Hoyo, M. (2021) · Revista Española de Cardiología · doi:10.1016/j.recesp.2021.05.017 · Cited by 59
Kuhn, A. N., van der Giet, M., Kuhlmann, M. K., Martus, P., Mielke, N., Ebert, N., & Schäeffner, E. (2020) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2020.09.015 · Cited by 59
CD4+ T cell and M2 macrophage infiltration predict dedifferentiated liposarcoma patient outcomes
Schroeder, B., LaFranzo, N. A., LaFleur, B., Gittelman, R., Vignali, M., Zhang, S., Flanagan, K. C., Rytlewski, J. A., Riolobos, L., Schulte, B. C., et al. (2021) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2021-002812 · Cited by 58
Prediction models for the incidence and progression of periodontitis: A systematic review
Du, M., Bo, T., Kapellas, K., & Peres, M. A. (2018) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13037 · Cited by 59
Cerqueira, D. D. C., Soares, C. M. B., Silva, V. R., Magalhães, J., de Barcelos, I. P., Duarte, M. G., Pinheiro, S. V. B., Colosimo, E. A., Silva, A. C. S. E., & OLIVEIRA, E. A. (2014) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.06630613 · Cited by 58
Risk prediction models of depression in older adults with chronic diseases
Zheng, Y., Zhang, C., & Liu, Y. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.05.078 · Cited by 57
Sekiguchi, M., Kakugawa, Y., Matsumoto, M., & Matsuda, T. (2018) · Journal of Gastroenterology · doi:10.1007/s00535-018-1433-7 · Cited by 57
Nochaiwong, S., Ruengorn, C., Koyratkoson, K., Thavorn, K., Awiphan, R., Chaisai, C., Phatthanasobhon, S., Noppakun, K., Suteeka, Y., Panyathong, S., et al. (2018) · Scientific Reports · doi:10.1038/s41598-018-33196-2 · Cited by 57
Clinical Prediction Models in Sports Medicine: A Guide for Clinicians and Researchers
Bullock, G. S., Hughes, T., Sergeant, J. C., Callaghan, M. J., Riley, R. D., & Collins, G. S. (2021) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2021.10697 · Cited by 57
Neuromarkers for Mental Disorders: Harnessing Population Neuroscience
Jollans, L., & Whelan, R. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00242 · Cited by 56
“Generally, the most efficient approach to biomarker development will take into consideration what we already know at every stage of the development pipeline (19).”
BackgroundInfluentialAortic Stiffness in a Mortality Risk Calculator for Kidney Transplant Recipients
Dahle, D. O., Eide, I. A., Åsberg, A., Leivestad, T., Holdaas, H., Jenssen, T. G., Fagerland, M. W., Pihlstrøm, H. K., Mjøen, G., & Hartmann, A. (2015) · Transplantation · doi:10.1097/tp.0000000000000660 · Cited by 56
Sandhu, P. K., Musaad, S. M. A., Remaley, A. T., Buehler, S. S., Strider, S., Derzon, J. H., Vesper, H. W., Ranne, A., Shaw, C. S., & Christenson, R. H. (2016) · The Journal of Applied Laboratory Medicine · doi:10.1373/jalm.2016.021006 · Cited by 55
“The utility of nontraditional markers in riskassessment isbestexaminedbycombining them with a model that includes traditional risk factors (8, 9).”
van Adrichem, R. A., Nelissen, R. G. H. H., Schipper, I. B., Rosendaal, F. R., & Cannegieter, S. C. (2015) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.12996 · Cited by 55
Performance of bleeding risk scores in dialysis patients
Ocak, G., Ramspek, C. L., Rookmaaker, M. B., Blankestijn, P. J., Verhaar, M. C., Bos, W. J. W., Dekker, F. W., & van Diepen, M. (2018) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfy387 · Cited by 55
Jabbarli, R., Reinhard, M., Roelz, R., Shah, M. J., Niesen, W., Kaier, K., Taschner, C. A., Weyerbrock, A., & Velthoven, V. V. (2015) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2015.81 · Cited by 54
Huang, X., Ribeiro, J. D., & Franklin, J. C. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00239 · Cited by 54
Samarakoon, U., Accarino, J. J. O., Wurcel, A. G., Jaggers, J. L., Judd, A. D., & Blumenthal, K. G. (2022) · Annals of Allergy Asthma & Immunology · doi:10.1016/j.anai.2022.12.023 · Cited by 54
Monteith, S., Glenn, T., Geddes, J., Whybrow, P. C., & Bauer, M. (2016) · International Journal of Bipolar Disorders · doi:10.1186/s40345-016-0051-7 · Cited by 50
Fracture Risk Assessment: State of the Art, Methodologically Unsound, or Poorly Reported?
Collins, G. S., & Michaëlsson, K. (2012) · Current Osteoporosis Reports · doi:10.1007/s11914-012-0108-1 · Cited by 51
Critical Assessment of Clinical Prognostic Tools in Melanoma
Mahar, A. L., Compton, C. C., Halabi, S., Hess, K. R., Gershenwald, J. E., Scolyer, R. A., & Groome, P. A. (2016) · Annals of Surgical Oncology · doi:10.1245/s10434-016-5212-5 · Cited by 51
Moik, F., Zöchbauer‐Müller, S., Posch, F., Pabinger, I., & Ay, C. (2020) · Cancers · doi:10.3390/cancers12061619 · Cited by 51
Jairam, P. M., Gondrie, M. J. A., Grobbee, D. E., Mali, W. P. T. M., Jacobs, P. C. A., van der Graaf, Y., & For the PROVIDI Study Group (2014) · Radiology · doi:10.1148/radiol.14132211 · Cited by 51
Self-citationStrutton, R., Du Chemin, A., Stratton, I. M., & Forster, A. S. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010952 · Cited by 38
Damen, J. A., Debray, T. P. A., Pajouheshnia, R., Reitsma, J. B., Scholten, R. J. P. M., Moons, K. G. M., & Hooft, L. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-026160 · Cited by 50
Self-citationAlblas, M., Velt, K., Pashayan, N., Widschwendter, M., Steyerberg, E. W., & Vergouwe, Y. (2018) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2018.03.023 · Cited by 50
Self-citationEmergence of digital biomarkers to predict and modify treatment efficacy: machine learning study
Guthrie, N. L., Carpenter, J., Edwards, K. L., Appelbaum, K. J., Dey, S., Eisenberg, D. M., Katz, D. L., & Berman, M. A. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-030710 · Cited by 50
Singh, P., Adderley, N. J., Hazlehurst, J., Price, M. J., Tahrani, A. A., Nirantharakumar, K., & Bellary, S. (2021) · Diabetes Care · doi:10.2337/dc21-0166 · Cited by 51
Amarasingham, R., Audet, A.-M. J., Bates, D. W., Cohen, I. G., Entwistle, M., Escobar, G. J., Liu, V., Etheredge, L., Lo, B., Ohno-Machado, L., et al. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1163 · Cited by 47
Fortgeschrittene Methoden zur Modellierung von diskreten Ereigniszeiten
Berger, M. (2023) · bonndoc (University of Bonn) · Cited by 50
Stynes, S. M., Konstantinou, K., Ogollah, R. O., Hay, E. M., & Dunn, K. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0191852 · Cited by 50
Validating Risk Prediction Models of Diabetes Remission After Sleeve Gastrectomy
Shen, S., Wang, W., Tam, K., Chen, H., Lin, Y., Wang, S.-Y., Huang, M., & Su, Y. (2018) · Obesity Surgery · doi:10.1007/s11695-018-3510-7 · Cited by 49
ROSNER, B. A., Colditz, G. A., Hankinson, S. E., Sullivan‐Halley, J., Lacey, J. V., & Bernstein, L. (2013) · Breast Cancer Research and Treatment · doi:10.1007/s10549-013-2719-3 · Cited by 49
Considerations in the reliability and fairness audits of predictive models for advance care planning
Lu, J., Sattler, A., Wang, S., Khaki, A. R., Callahan, A., Fleming, S., Fong, R., Ehlert, B., Li, R. C., Shieh, L., et al. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.943768 · Cited by 40
Yatsuya, H., Iso, H., Li, Y., Yamagishi, K., Kokubo, Y., Saito, I., Sawada, N., Inoue, M., & Tsugane, S. (2016) · Circulation Journal · doi:10.1253/circj.cj-16-0081 · Cited by 47
Halicka, M., Wilby, M. J., Duarte, R. V., & Brown, C. A. (2023) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-023-06446-2 · Cited by 49
Tousignant‐Laflamme, Y., Houle, C., Cook, C. E., Naye, F., LeBlanc, A., & Décary, S. (2022) · Physical Therapy · doi:10.1093/ptj/pzac023 · Cited by 47
Alba, A. C., Braga, J. R., Gewarges, M., Walter, S. D., Guyatt, G. H., & Ross, H. J. (2013) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2013.09.024 · Cited by 47
Smith, M., Barber, P. A., Scrivener, B. J., & Stinear, C. M. (2022) · Neurorehabilitation and neural repair · doi:10.1177/15459683221085287 · Cited by 46
Inneh, I. A., Lewis, C. G., & Schutzer, S. F. (2014) · The Journal of Arthroplasty · doi:10.1016/j.arth.2014.05.007 · Cited by 45
Schiefer, S., Wirsik, N. M., Kalkum, E., Seide, S., Nienhüser, H., Müller-Stich, B. P., Billeter, A. T., Büchler, M. W., Schmidt, T., & Probst, P. (2022) · Diagnostics · doi:10.3390/diagnostics12030593 · Cited by 45
Intracerebral Hematoma Due to Aneurysm Rupture
Jabbarli, R., Reinhard, M., Roelz, R., Shah, M. J., Niesen, W., Kaier, K., Taschner, C. A., Weyerbrock, A., & Velthoven, V. V. (2015) · Neurosurgery · doi:10.1227/neu.0000000000001136 · Cited by 45
Accelerated and Interpretable Oblique Random Survival Forests
Jaeger, B. C., Welden, S., Lenoir, K. M., Speiser, J. L., Segar, M. W., Pandey, A., & Pajewski, N. M. (2023) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2023.2231048 · Cited by 44
Prognostic DNA methylation markers for sporadic colorectal cancer: a systematic review
Draht, M. X. G., Goudkade, D., Koch, A., Grabsch, H. I., Weijenberg, M. P., van Engeland, M., Melotte, V., & Smits, K. M. (2018) · Clinical Epigenetics · doi:10.1186/s13148-018-0461-8 · Cited by 45
Allotey, J., Fernandez-Felix, B. M., Zamora, J., Moss, N., Bagary, M. S., Kelso, A. R. C., Khan, R., van der Post, J., Mol, B. W., Pirie, A. M., et al. (2019) · PLoS Medicine · doi:10.1371/journal.pmed.1002802 · Cited by 44
Jaffray, J., Branchford, B. R., Goldenberg, N. A., Malvar, J., Croteau, S. E., Silvey, M., Fargo, J. H., Cooper, J. D., Bakeer, N., Sposto, R., et al. (2020) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2020.09.016 · Cited by 44
Metadata
- Authors
- Moons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E.
- Affiliations
- University Medical Center Utrecht (NL)
- The University of Sydney (AU)
- MRC Clinical Trials Unit at UCL (GB)
- University of Oxford (GB)
- Corresponding author
- Moons, Karel G M
- Venue
- Heart
- Publisher
- BMJ
- Published
- 2012-03-07
- Volume
- 98
- Issue
- 9
- Pages
- 683-690
- ISSN
- 1355-6037 (ISSN-L)
- 1468-201X
- OpenAlex
- W2146404132
- Cited by
- 889
- References
- 100
- Citations by year
- 2026: 42 · 2025: 52 · 2024: 59 · 2023: 59 · 2022: 92 · 2021: 87 · 2020: 75 · 2019: 46 · 2018: 76 · 2017: 53 · 2016: 58 · 2015: 68 · 2014: 56 · 2013: 44 · 2012: 18
- FWCI
- 9.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Heart (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Medical Research Council (MC_EX_G0800814)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- risk prediction model, model development, internal validation, optimism bias, cardiovascular risk prediction, predictive performance, clinical decision-making, cost-effectiveness
- MeSH terms
- Cardiovascular Diseases, Cost-Benefit Analysis, Decision Making, Decision Support Techniques, Humans, Models, Theoretical, Predictive Value of Tests, Risk, Reproducibility of Results, Biomarkers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.