Prognosis and prognostic research: Developing a prognostic model
Abstract
In the second article in their series, Patrick Royston and colleagues describe different approaches to building clinical prognostic models
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The TRIPOD Statement covers studies that report solely development, both development and external validation, and solely external validation (with or without updating), of a prediction model.
“The TRIPOD Statement covers studies that report solely development [12,15], both development and external validation, and solely external validation (with or without updating), of a prediction model [14] (Figure 1).”
From Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement (2015) · cites this paper as [12]
Use of poor methods may lead to biased results.
“Use of poor methods may lead to biased results [2],[7],[10],[12],[13],[15],[17]–[32].”
From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [7]
The TRIPOD Statement covers studies that report solely development, both development and external validation, and solely external validation (with or without updating), of a prediction model.
“The TRIPOD Statement covers studies that report solely development,12,15 both development and external validation, and solely external validation (with or without updating), of a prediction model14 (Figure 3).”
From Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement (2015) · cites this paper as [12]
Cites
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,384 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493 · see citing works
The cost of dichotomising continuous variables
Altman, D. G., & Royston, P. (2006) · BMJ · doi:10.1136/bmj.332.7549.1080 · Cited by 2,688 · see citing works
Self-citationReview: a gentle introduction to imputation of missing values.
Donders, A. R. T., van der Heijden, G. J. M. G., Stijnen, T., & Moons, K. G. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.014 · Cited by 2,582 · see citing works
Self-citationDichotomizing continuous predictors in multiple regression: a bad idea
Royston, P., Altman, D. G., & Sauerbrei, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2331 · Cited by 2,312 · see citing works
Self-citationPrognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436 · see citing works
Self-citationPrognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337 · see citing works
Self-citationGreenland, P. (2007) · Statistics in Medicine · doi:10.1002/sim.2976 · Cited by 1,300 · see citing works
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934 · see citing works
Self-citationInappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis
Sun, G.-W., Shook, T. L., & Kay, G. L. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(96)00025-x · Cited by 933 · cited 1× · see citing works
“However, predictors that are not significant in univariable analysis should not be excluded as candidates.”
BackgroundBootstrap Methods for Developing Predictive Models
Austin, P. C., & Tu, J. V. (2004) · The American Statistician · doi:10.1198/0003130043277 · Cited by 622 · see citing works
Griffiths, G. O. (1999) · The Lancet · doi:10.1016/s0140-6736(98)03544-2 · Cited by 602 · see citing works
Royston, P. J., & Sauerbrei, W. (2008) · Cited by 601 · see citing works
Self-citationPrognostic Modeling with Logistic Regression Analysis
Steyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2001) · Medical Decision Making · doi:10.1177/0272989x0102100106 · Cited by 559 · see citing works
Sauerbrei, W., & Royston, P. J. (1999) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/1467-985x.00122 · Cited by 475 · see citing works
Self-citationMultivariable Analysis: A Primer for Readers of Medical Research
Katz, M. H. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-8-200304150-00012 · Cited by 400 · see citing works
The Use of Resampling Methods to Simplify Regression Models in Medical Statistics
Sauerbrei, W. (1999) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00155 · Cited by 310 · see citing works
Why Stepdown Procedures in Variable Selection
Mantel, N. (1970) · Technometrics · doi:10.1080/00401706.1970.10488701 · Cited by 275 · see citing works
Statistical Methods in Epidemiology
Boucher, K. M., Slattery, M. L., Berry, T. D., Quesenberry, C. P., & Anderson, K. N. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00129-2 · Cited by 237 · see citing works
Building Multivariable Regression Models with Continuous Covariates in Clinical Epidemiology
Sauerbrei, W., & Royston, P. J. (2005) · Methods of Information in Medicine · doi:10.1055/s-0038-1634008 · Cited by 198 · see citing works
Self-citationExplained Variance in Logistic Regression
DeMaris, A. (2002) · Sociological Methods & Research · doi:10.1177/0049124102031001002 · Cited by 142 · see citing works
Assessing the applicability of scoring systems for predicting postoperative nausea and vomiting
Bosch, J. E. V. D., Kalkman, C. J., Vergouwe, Y., van Klei, W. A., Bonsel, G. J., Grobbee, D. E., & Moons, K. G. M. (2005) · Anaesthesia · doi:10.1111/j.1365-2044.2005.04121.x · Cited by 112 · see citing works
Self-citationRosenberg, P. S., Katki, H. A., Swanson, C. A., et al. (2003) · Statistics in Medicine · doi:10.1002/sim.1638 · Cited by 60 · see citing works
Kattan, M. W. (2006) · Cancer · doi:10.1002/cncr.22314 · Cited by 34 · see citing works
Influence of model‐building strategies on the results of a case‐control study
Blettner, M., & Sauerbrei, W. (1993) · Statistics in Medicine · doi:10.1002/sim.4780121405 · Cited by 30 · see citing works
Reporting of Predictive Logistic Models Should Be Based on Evidence-Based Guidelines
Hernández, A. V., Vergouwe, Y., & Steyerberg, E. W. (2003) · CHEST Journal · doi:10.1378/chest.124.5.2034 · Cited by 5 · see citing works
Self-citationHarrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Flagged first, then by citations
Cited by
Sorted by citations
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388 · see citing works
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267 · see citing works
Self-citationPROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984 · see citing works
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · see citing works
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927 · see citing works
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669 · see citing works
Self-citationPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Self-citationPrognosis and prognostic research: validating a prognostic model
Altman, D. G., Vergouwe, Y., Royston, P., & Moons, K. G. M. (2009) · BMJ · doi:10.1136/bmj.b605 · Cited by 1,436 · see citing works
Self-citationPrognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337 · see citing works
Self-citationRiley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
Self-citationRisk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195 · see citing works
Self-citationVariable selection strategies and its importance in clinical prediction modelling.
Chowdhury, M. Z. I., & Turin, T. C. (2020) · Family Medicine and Community Health · doi:10.1136/fmch-2019-000262 · Cited by 1,011 · see citing works
Prognosis and prognostic research: application and impact of prognostic models in clinical practice
Moons, K. G. M., Altman, D. G., Vergouwe, Y., & Royston, P. (2009) · BMJ · doi:10.1136/bmj.b606 · Cited by 934 · see citing works
Self-citationPrediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935 · see citing works
Self-citationMoons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E. (2012) · Heart · doi:10.1136/heartjnl-2011-301246 · Cited by 889 · see citing works
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Circulation · doi:10.1161/circulationaha.114.014508 · Cited by 793 · see citing works
Self-citationRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · see citing works
Self-citationReporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · see citing works
Self-citationUdo, E. O., Zuithoff, N. P. A., van Hemel, N. M., de Cock, C. C., Hendriks, T., Doevendans, P. A., & Moons, K. G. M. (2012) · Heart Rhythm · doi:10.1016/j.hrthm.2011.12.014 · Cited by 597 · see citing works
Self-citationRisk models and scores for type 2 diabetes: systematic review
Noble, D., Mathur, R., Dent, T., Meads, C., & Greenhalgh, T. (2011) · BMJ · doi:10.1136/bmj.d7163 · Cited by 551 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Urology · doi:10.1016/j.eururo.2014.11.025 · Cited by 439 · see citing works
Self-citationGeersing, G.-J., Bouwmeester, W., Zuithoff, P., Spijker, R., Leeflang, M., & Moons, K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0032844 · Cited by 350 · see citing works
Self-citationVistisen, D., Andersen, G. S., Hansen, C. S., Hulman, A., Henriksen, J. E., Bech-Nielsen, H., & Jørgensen, M. E. (2016) · Circulation · doi:10.1161/circulationaha.115.018844 · Cited by 229 · see citing works
Diagnostic and prognostic prediction models
Hendriksen, J. M. T., Geersing, G. J., Moons, K. G. M., & de Groot, J. A. H. (2013) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.12262 · Cited by 229 · see citing works
Self-citationReporting methods in studies developing prognostic models in cancer: a review
Mallett, S., Royston, P., Dutton, S., Waters, R., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-20 · Cited by 206 · see citing works
“We assessed articles in random order using a piloted data extraction form based on key aspects of model design and development from the current literature [1,3,4,6,7].”
BackgroundMethodInfluentialSelf-citationKarter, A. J., Warton, E. M., Lipska, K. J., Ralston, J. D., Moffet, H. H., Jackson, G. G., Huang, E. S., & Miller, D. R. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.3844 · Cited by 150 · see citing works
Different definitions in childhood asthma: how dependable is the dependent variable?
Van Wonderen, K. E., Van Der Mark, L. B., Mohrs, J., Bindels, P. J. E., Van Aalderen, W. M. C., & Ter Riet, G. (2010) · European Respiratory Journal · doi:10.1183/09031936.00154409 · Cited by 100 · see citing works
Portegijs, E., Buurman, B. M., Essink-Bot, M.-L., Zwinderman, A. H., & de Rooij, S. E. (2012) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2012.04.003 · Cited by 88 · see citing works
Ensor, J., Riley, R. D., Moore, D., Snell, K. I. E., Bayliss, S., & Fitzmaurice, D. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011190 · Cited by 84 · see citing works
Sanfelici, R., Dwyer, D. B., Antonucci, L. A., & Koutsouleris, N. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.02.009 · Cited by 82 · see citing works
Use of reclassification for assessment of improved prediction: an empirical evaluation
Tzoulaki, I., Liberopoulos, G., & Ioannidis, J. P. A. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr013 · Cited by 70 · see citing works
Alzheimer's disease digital biomarkers multidimensional landscape and AI model scoping review
Qi, W., Zhu, X., Wang, B., Shi, Y., Dong, C., Shen, S., Li, J., Zhang, K., He, Y., Zhao, M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01640-z · Cited by 78 · see citing works
Witlox, L., Velthuis, M. J., Boer, J. H., Steins Bisschop, C. N., Wall, E. V. D., Meulen, W. J. T. M. V. D., Schröder, C. D., Peeters, P. H. M., & May, A. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215517 · Cited by 63 · see citing works
Fracture Risk Assessment: State of the Art, Methodologically Unsound, or Poorly Reported?
Collins, G. S., & Michaëlsson, K. (2012) · Current Osteoporosis Reports · doi:10.1007/s11914-012-0108-1 · Cited by 51 · see citing works
Wang, J. L., Patten, S., Sareen, J., Bolton, J., Schmitz, N., & MacQueen, G. (2014) · Depression and Anxiety · doi:10.1002/da.22215 · Cited by 43 · see citing works
Predicting and preventing acute care re-utilization by patients with diabetes
Rubin, D. J., & Shah, A. A. (2021) · Current Diabetes Reports · doi:10.1007/s11892-021-01402-7 · Cited by 44 · see citing works
Postpartum depression: a developed and validated model predicting individual risk in new mothers
Munk-Olsen, T., Liu, X., Madsen, K. B., Kjeldsen, M.-M. Z., Petersen, L. V., Bergink, V., Skalkidou, A., Vigod, S. N., Frokjaer, V. G., Pedersen, C. B., et al. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-02190-8 · Cited by 36 · see citing works
Predicting Non-response to Ranibizumab in Patients with Neovascular Age-related Macular Degeneration
van Asten, F., Rovers, M. M., Lechanteur, Y. T. E., Smailhodzic, D., Muether, P. S., Chen, J., den Hollander, A. I., Fauser, S., Hoyng, C. B., van der Wilt, G. J., et al. (2014) · Ophthalmic Epidemiology · doi:10.3109/09286586.2014.949010 · Cited by 37 · see citing works
Jinks, R. C., Royston, P., & Parmar, M. K. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0078-y · Cited by 32 · see citing works
Self-citationPosch, F., Gebhart, J., Rand, J. H., Koder, S., Quehenberger, P., Pengo, V., Ay, C., & Pabinger, I. (2017) · BMC Medicine · doi:10.1186/s12916-017-0807-7 · Cited by 25 · see citing works
Clinical prediction models using machine learning in oncology: challenges and recommendations
Collins, G. S., Chester-Jones, M., Gerry, S., Ma, J., Matos, J., Sehjal, J., Tsegaye, B., & Dhiman, P. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000914 · Cited by 27 · see citing works
Predictors of Success for Pelvic Floor Muscle Training in Pelvic Organ Prolapse
Wiegersma, M., Panman, C. M. C. R., Hesselink, L. C., Malmberg, A. G. A., Berger, M. Y., Kollen, B. J., & Dekker, J. H. (2019) · Physical Therapy · doi:10.1093/ptj/pzy114 · Cited by 17 · see citing works
- Authors
- Royston, P., Moons, K. G. M., Altman, D. G., & Vergouwe, Y.
- Affiliations
- MRC Clinical Trials Unit at UCL (GB)
- University Medical Center Utrecht (NL)
- University of Oxford (GB)
- Corresponding author
- Royston, P.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2009-03-31
- Volume
- 338
- Issue
- mar31 1
- Pages
- b604-b604
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- OpenAlex
- W2042080283
- Cited by
- 1,172
- References
- 31
- Citations by year
- 2026: 37 · 2025: 52 · 2024: 51 · 2023: 71 · 2022: 87 · 2021: 87 · 2020: 98 · 2019: 77 · 2018: 88 · 2017: 87 · 2016: 57 · 2015: 83 · 2014: 79 · 2013: 67 · 2012: 57
- FWCI
- 41.88
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- BMJ (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Cancer Research UK
- Medical Research Council (MC_EX_G0800814)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- prognostic model, prognostic research, clinical prediction model, model development, prognostic factors
- MeSH terms
- Data Collection, Humans, Kidney Neoplasms, Prognosis, Regression Analysis, ROC Curve, Multivariate Analysis, Risk Assessment, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
