Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
Prediction models are increasingly used to complement clinical reasoning and decision making in modern medicine in general, and in the cardiovascular domain in particular. Developed models first and foremost need to provide accurate and (internally and externally) validated estimates of probabilities of specific health conditions or outcomes in targeted patients. The adoption of such models must guide physician's decision making and an individual's behaviour, and consequently improve individual outcomes and the cost-effectiveness of care. In a series of two articles we review the consecutive steps generally advocated for risk prediction model research. This first article focuses on the different aspects of model development studies, from design to reporting, how to estimate a model's predictive performance and the potential optimism in these estimates using internal validation techniques, and how to quantify the added or incremental value of new predictors or biomarkers (of whatever type) to existing predictors. Each step is illustrated with empirical examples from the cardiovascular field.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is therefore essential that key details of how a prediction model was developed and validated be clearly reported to enable synthesis and critical appraisal of all relevant information (14,(33)(34)(35)(36).”
From Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [14]
In the citing paper
“The TRIPOD Statement covers studies that report solely development [12,15], both development and external validation, and solely external validation (with or without updating), of a prediction model [14] (Figure 1).”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [15]
In the citing paper
“The TRIPOD Statement covers studies that report solely development,12,15 both development and external validation, and solely external validation (with or without updating), of a prediction model14 (Figure 3).”
From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [15]
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Prediction of coronary heart disease using risk factor categories
Wilson, P. W. F., D’Agostino, R. B., Levy, D., Belanger, A. M., Silbershatz, H., & Kannel, W. B. (1998) · Circulation · doi:10.1161/01.cir.97.18.1837 · Cited by 9,581
Harrell, F. E. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,195
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · cited 3×
“For example, the net reclassification improvement (NRI) does this by quantifying the number of individuals that are correctly reclassified into clinically meaningful higher or lower risk categories with the addition of a new predictor, using pre-specified risk groups.(84) Correct reclassifications are shifts to a high…”
BackgroundMethodInfluentialDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Analysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012
Levey, A. S., Coresh, J., Greene, T., Stevens, L. A., Zhang, Y. (., Hendriksen, S., Kusek, J. W., & Van Lente, F. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-145-4-200608150-00004 · Cited by 5,725
Analysis of Incomplete Multivariate Data
Booth, D. E. (2000) · Technometrics · doi:10.1080/00401706.2000.10486013 · Cited by 5,627
Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project
Conroy, R. (2003) · European Heart Journal · doi:10.1016/s0195-668x(03)00114-3 · Cited by 5,290
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
European system for cardiac operative risk evaluation (EuroSCORE)
Nashef, S. A. M., Roques, F., Michel, P., Gauducheau, E., Lemeshow, S., & Salamon, R. (1999) · European Journal of Cardio-Thoracic Surgery · doi:10.1016/s1010-7940(99)00134-7 · Cited by 3,362
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
The cost of dichotomising continuous variables
Altman, D. G., & Royston, P. (2006) · BMJ · doi:10.1136/bmj.332.7549.1080 · Cited by 2,688
Self-citationReview: a gentle introduction to imputation of missing values.
Donders, A. R. T., van der Heijden, G. J. M. G., Stijnen, T., & Moons, K. G. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.014 · Cited by 2,582
Self-citationExtensions of net reclassification improvement calculations to measure usefulness of new biomarkers
Pencina, M. J., D'Agostino, R. B., & Steyerberg, E. W. (2011) · Statistics in Medicine · doi:10.1002/sim.4085 · Cited by 2,453
Dichotomizing continuous predictors in multiple regression: a bad idea
Royston, P., Altman, D. G., & Sauerbrei, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2331 · Cited by 2,312
Self-citationPeduzzi, P., Concato, J., Feinstein, A. R., & Holford, T. R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00048-8 · Cited by 2,154 · cited 2×
“< A total of 473 major cardiovascular events were recorded during the follow-up, yielding 18 events per candidate predictor.(44)”
BackgroundModelling for Survival Data in Medical Research
(1995) · Technometrics · doi:10.1080/00401706.1995.10485920 · Cited by 2,145
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,112
Patel, A. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61303-8 · Cited by 2,054
Self-citationRegression modelling strategies for improved prognostic prediction
Harrell, F. E., Lee, K. L., Califf, R. M., Pryor, D. B., & Rosati, R. A. (1984) · Statistics in Medicine · doi:10.1002/sim.4780030207 · Cited by 1,798
Presentation of multivariate data for clinical use: The Framingham Study risk score functions
Sullivan, L. M., Massaro, J. M., & D'Agostino, R. B. (2004) · Statistics in Medicine · doi:10.1002/sim.1742 · Cited by 1,753
Assmann, G., Cullen, P., & Schulte, H. (2002) · Circulation · doi:10.1161/hc0302.102575 · Cited by 1,541
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
Prognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436
Self-citationClinical Prediction Rules: Applications and Methodological Standards
Wasson, J. H., Sox, H. C., Neff, R. K., & Goldman, L. (1985) · New England Journal of Medicine · doi:10.1056/nejm198509263131306 · Cited by 1,357
Prognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337
Self-citationGreenland, P. (2008) · Statistics in Medicine · doi:10.1002/sim.2976 · Cited by 1,300
Value of assessment of pretest probability of deep-vein thrombosis in clinical management
Wells, P. S., Anderson, D. R., Bormanis, J., Guy, F., Mitchell, M., Gray, L., Clement, C., Robinson, K. S., & Lewandowski, B. (1997) · The Lancet · doi:10.1016/s0140-6736(97)08140-3 · Cited by 1,229
Risk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195
Self-citationPrognosis and prognostic research: Developing a prognostic model
Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y. (2009) · BMJ · doi:10.1136/bmj.b604 · Cited by 1,172
Self-citationHlatky, M. A., Greenland, P., Arnett, D. K., Ballantyne, C. M., Criqui, M. H., Elkind, M. S. V., Go, A. S., Harrell, F. E., Hong, Y., Howard, B. V., et al. (2009) · Circulation · doi:10.1161/circulationaha.109.192278 · Cited by 1,139
The Risk of Determining Risk with Multivariable Models
Concato, J., Feinstein, A. R., & Holford, T. R. (1993) · Annals of Internal Medicine · doi:10.7326/0003-4819-118-3-199302010-00009 · Cited by 1,115
Using the outcome for imputation of missing predictor values was preferred
Moons, K. G. M., Donders, R. A. R. T., Stijnen, T., & Harrell, F. E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.009 · Cited by 994
Self-citationPrognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934
Self-citationInappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis
Sun, G.-W., Shook, T. L., & Kay, G. L. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(96)00025-x · Cited by 933
Laupacis, A. (1997) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1997.03540300056034 · Cited by 793
Derksen, S., & Keselman, H. J. (1992) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1992.tb00992.x · Cited by 771
Cardiovascular Risk Prediction: Basic Concepts, Current Status, and Future Directions
Lloyd-Jones, D. M. (2010) · Circulation · doi:10.1161/circulationaha.109.849166 · Cited by 705
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691
Concordance probability and discriminatory power in proportional hazards regression
Gönen, M., & Heller, G. (2005) · Biometrika · doi:10.1093/biomet/92.4.965 · Cited by 673
Woodward, M., Brindle, P., & Tunstall-Pedoe, H. (2007) · Heart · doi:10.1136/hrt.2006.108167 · Cited by 667
Self-citationMissing covariate data in medical research: To impute is better than to ignore
Janssen, K. J. M., Donders, A. R. T., Harrell, F. E., Vergouwe, Y., Chen, Q., Grobbee, D. E., & Moons, K. G. M. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.12.008 · Cited by 592
Self-citationStatistical aspects of prognostic factor studies in oncology
Simon, R., & Altman, D. (1994) · British Journal of Cancer · doi:10.1038/bjc.1994.192 · Cited by 582
Self-citationValidation, updating and impact of clinical prediction rules: A review
Toll, D. B., Janssen, K. J. M., Vergouwe, Y., & Moons, K. G. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.04.008 · Cited by 580
Self-citationSteyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2001) · Medical Decision Making · doi:10.1177/0272989x0102100106 · Cited by 550
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 463
Systematic review of prognostic models in traumatic brain injury
Perel, P., Edwards, P., Wentz, R., & Roberts, I. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-38 · Cited by 344
Evaluation of the Performance of Survival Analysis Models: Discrimination and Calibration Measures
D'Agostino, R. B., & Nam, B.-H. (2003) · Handbook of Statistics · doi:10.1016/s0169-7161(03)23001-7 · Cited by 338
Estimation of time‐dependent area under the ROC curve for long‐term risk prediction
Chambless, L. E., & Diao, G. (2006) · Statistics in Medicine · doi:10.1002/sim.2299 · Cited by 311
Integrating the Predictiveness of a Marker with Its Performance as a Classifier
Pepe, M. S., Feng, Z., Huang, Y., Longton, G., Prentice, R., Thompson, I. M., & Zheng, Y. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwm305 · Cited by 294
Nozaki, T., Sugiyama, S., Koga, H., Sugamura, K., Ohba, K., Matsuzawa, Y., Sumida, H., Matsui, K., Jinnouchi, H., & Ogawa, H. (2009) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2009.05.022 · Cited by 293
Clinical Epidemiology: Principles, Methods, And Applications For Clinical Research
Grobbee, D. E., & Hoes, A. W. (2008) · Cited by 289
Self-citationWhy Stepdown Procedures in Variable Selection
Mantel, N. (1970) · Technometrics · doi:10.1080/00401706.1970.10488701 · Cited by 280
Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score
Tzoulaki, I. (2009) · JAMA · doi:10.1001/jama.2009.1757 · Cited by 276
Paynter, N. P., Chasman, D. I., Buring, J. E., Shiffman, D., Cook, N. R., & Ridker, P. M. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-150-2-200901200-00003 · Cited by 262
Systematic Review of Prognostic Models in Patients with Acute Stroke
Counsell, C., & Dennis, M. (2001) · Cerebrovascular Diseases · doi:10.1159/000047699 · Cited by 250
May, S. J., & Hosmer, D. W. (1998) · Lifetime Data Analysis · doi:10.1023/a:1009612305785 · Cited by 227
Clinical Judgment Revisited: The Distraction of Quantitative Models
Feinstein, A. R. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-9-199405010-00012 · Cited by 221
Reporting methods in studies developing prognostic models in cancer: a review
Mallett, S., Royston, P., Dutton, S., Waters, R., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-20 · Cited by 206
Self-citationReporting performance of prognostic models in cancer: a review.
Mallett, S., Royston, P., Waters, R., Dutton, S., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-21 · Cited by 195
Self-citationContemporary model for cardiovascular risk prediction in people with type 2 diabetes
Kengne, A. P., Patel, A., Marre, M., Travert, F., Lievre, M., Zoungas, S., Chalmers, J., Colagiuri, S., Grobbee, D. E., Hamet, P., et al. (2011) · European Journal of Cardiovascular Prevention & Rehabilitation · doi:10.1177/1741826710394270 · Cited by 183
Self-citationMoons, K. G. M., Donders, A. R. T., Steyerberg, E. W., & Harrell, F. E. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.01.020 · Cited by 178
Self-citationPrediction models in reproductive medicine: a critical appraisal†
Leushuis, E., van der Steeg, J. W., Steures, P., Bossuyt, P. M. M., Eijkemans, M. J. C., van der Veen, F., Mol, B. W. J., & Hompes, P. G. A. (2009) · Human Reproduction Update · doi:10.1093/humupd/dmp013 · Cited by 173
Dealing with Missing Predictor Values When Applying Clinical Prediction Models
Janssen, K. J. M., Vergouwe, Y., Donders, A. R. T., Harrell, F. E., Chen, Q., Grobbee, D. E., & Moons, K. G. M. (2009) · Clinical Chemistry · doi:10.1373/clinchem.2008.115345 · Cited by 168
Self-citationRansohoff, D. F. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.04.020 · Cited by 163
Predicting Clinical States in Individual Patients
Braitman, L. E., & Davidoff, F. (1996) · Annals of Internal Medicine · doi:10.7326/0003-4819-125-5-199609010-00008 · Cited by 157
Christensen, E. (2004) · Journal of Hepatology · doi:10.1016/j.jhep.2004.06.005 · Cited by 147
Prognostic Models: A Methodological Framework and Review of Models for Breast Cancer
Altman, D. G. (2009) · Cancer Investigation · doi:10.1080/07357900802572110 · Cited by 142
Self-citationAdvantages of the nested case-control design in diagnostic research
Biesheuvel, C. J., Vergouwe, Y., Oudega, R., Hoes, A. W., Grobbee, D. E., & Moons, K. G. M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-48 · Cited by 139
Self-citationTest Research versus Diagnostic Research
Moons, K. G. M., Biesheuvel, C. J., & Grobbee, D. E. (2004) · Clinical Chemistry · doi:10.1373/clinchem.2003.024752 · Cited by 138
Self-citationCategorising continuous variables
Altman, D. G. (1991) · British Journal of Cancer · doi:10.1038/bjc.1991.441 · Cited by 130
Self-citationde Kleijn, D. P. V., Moll, F. L., Hellings, W. E., Ozsarlak‐Sozer, G., de Bruin, P. C., Doevendans, P. A., Vink, A., Catanzariti, L. M., Schoneveld, A. H., Algra, A., et al. (2009) · Arteriosclerosis Thrombosis and Vascular Biology · doi:10.1161/atvbaha.109.194944 · Cited by 115
Redundancy of Single Diagnostic Test Evaluation
Moons, K. G. M., van Es, G.-A., Michel, B. C., Buller, H. R., Habbema, J. D. F., & Grobbee, D. E. (1999) · Epidemiology · doi:10.1097/00001648-199905000-00015 · Cited by 98
Self-citationIoannidis, J. P. A. (2009) · Circulation Cardiovascular Genetics · doi:10.1161/circgenetics.108.833392 · Cited by 91
Criteria for Scientific Evaluation of Novel Markers: A Perspective
Moons, K. G. M. (2010) · Clinical Chemistry · doi:10.1373/clinchem.2009.134155 · Cited by 78
Self-citationWhen should we remain blind and when should our eyes remain open in diagnostic studies?
Moons, K. G. M., & Grobbee, D. E. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00408-0 · Cited by 75
Self-citationOverview of Risk Prediction Models in Cardiovascular Disease Research
Cui, J. S. (2009) · Annals of Epidemiology · doi:10.1016/j.annepidem.2009.05.005 · Cited by 68
Agreement of Self-Reported Use of Menopausal Hormone Replacement Therapy with Physician Reports
Jain, M. G., Rohan, T. E., & Howe, G. R. (1999) · Epidemiology · doi:10.1097/00001648-199905000-00012 · Cited by 66
Multiple imputation to correct for partial verification bias revisited
de Groot, J. A. H., Janssen, K. J. M., Zwinderman, A. H., Moons, K. G. M., & Reitsma, J. B. (2008) · Statistics in Medicine · doi:10.1002/sim.3410 · Cited by 52
Self-citationWhy Stepdown Procedures in Variable Selection
Mantel, N. (1970) · Technometrics · doi:10.2307/1267207 · Cited by 47
Modeling the effects of continuous risk factors.
Royston, P., Sauerbrei, W., & Altman, D. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00163-8 · Cited by 44
Self-citationWatts, G. (2009) · BMJ · doi:10.1136/bmj.b1317 · Cited by 38
Strengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration
Janssens, A. C. J. W., Ioannidis, J. P. A., Bedrosian, S. R., Boffetta, P., Dolan, S. M., Dowling, N. F., Fortier, I., Freedman, A. N., Grimshaw, J., Gulcher, J. R., et al. (2011) · European Journal of Human Genetics · doi:10.1038/ejhg.2011.27 · Cited by 34
Strengthening the reporting of genetic risk prediction studies: the GRIPS statement
for the GRIPS Group, Janssens, A. C. J. W., Ioannidis, J. P. A., van Duijn, C. M., Little, J., & Khoury, M. J. (2011) · European Journal of Human Genetics · doi:10.1038/ejhg.2011.25 · Cited by 21
Strengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration
Janssens, A. C. J. W., Ioannidis, J. P. A., Bedrosian, S. R., Boffetta, P., Dolan, S. M., Dowling, N. F., Fortier, I., Freedman, A. N., Grimshaw, J., Gulcher, J. R., et al. (2011) · European Journal of Epidemiology · doi:10.1007/s10654-011-9551-z · Cited by 19
Strengthening the reporting of genetic risk prediction studies: the GRIPS statement
for the GRIPS Group, Janssens, A. C. J. W., Ioannidis, J. P. A., van Duijn, C. M., Little, J., & Khoury, M. J. (2011) · European Journal of Epidemiology · doi:10.1007/s10654-011-9552-y · Cited by 16
Prognostic Models: A Methodological Framework and Review of Models for Breast Cancer
Altman, D. G. (2007) · CRC Press eBooks · doi:10.3109/9781420019940.002 · Cited by 15
Self-citationBreast Cancer: Translational Therapeutic Strategies
(2007) · CRC Press eBooks · doi:10.3109/9781420019940 · Cited by 9
Multiple imputation to correct for partial verification bias revisited (5880–5889)
de Groot, J. A. H., Janssen, K. J. M., Zwinderman, A. H., Moons, K. G. M., & Reitsma, J. B. (2008) · Statistics in Medicine · doi:10.1002/sim.3509 · Cited by 1
Self-citationHarrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Research works printed but not linked: 2
- Wells, 1997, A simple clinical model to categorize pretest probability in patients with suspected pulmonary embolism, Blood, 90, 424
- Hosmer DW Lemeshow S . Applied Survival Analysis: Regression Modelling of Time to Event Data. New York: Wiley, 1999.
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
Self-citation
Metadata
- Authors
- Moons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E.
- Affiliations
- University Medical Center Utrecht (NL)
- The University of Sydney (AU)
- MRC Clinical Trials Unit at UCL (GB)
- University of Oxford (GB)
- Corresponding author
- Moons, Karel G M
- Venue
- Heart
- Publisher
- BMJ
- Published
- 2012-03-07
- Volume
- 98
- Issue
- 9
- Pages
- 683-690
- ISSN
- 1355-6037 (ISSN-L)
- 1468-201X
- OpenAlex
- W2146404132
- Cited by
- 889
- References
- 100
- Citations by year
- 2026: 42 · 2025: 52 · 2024: 59 · 2023: 59 · 2022: 92 · 2021: 87 · 2020: 75 · 2019: 46 · 2018: 76 · 2017: 53 · 2016: 58 · 2015: 68 · 2014: 56 · 2013: 44 · 2012: 18
- FWCI
- 9.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Heart (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Medical Research Council (MC_EX_G0800814)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- risk prediction model, model development, internal validation, optimism bias, cardiovascular risk prediction, predictive performance, clinical decision-making, cost-effectiveness
- MeSH terms
- Cardiovascular Diseases, Cost-Benefit Analysis, Decision Making, Decision Support Techniques, Humans, Models, Theoretical, Predictive Value of Tests, Risk, Reproducibility of Results, Biomarkers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.