CoChat
Sign Up
Cited by 271Statistical Methods in EpidemiologyMeta-analysis and systematic reviewsArtificial Intelligence in Healthcare

Development and Reporting of Prediction Models: Guidance for Authors From Editors of Respiratory, Sleep, and Critical Care Journals

Open access
Leisman, Daniel E., Harhay, Michael O., Lederer, David J., Abramson, Michael, Adjei, Alex A., Bakker, Jan, Ballas, Zuhair K., Barreiro, Esther +31 more·2020·Critical Care Medicine
DOI 10.1097/ccm.0000000000004246PMC PMC7161722
Open this paper in CoChat — free account

Abstract

Prediction models aim to use available data to predict a health state or outcome that has not yet been observed. Prediction is primarily relevant to clinical practice, but is also used in research, and administration. While prediction modeling involves estimating the relationship between patient factors and outcomes, it is distinct from casual inference. Prediction modeling thus requires unique considerations for development, validation, and updating. This document represents an effort from editors at 31 respiratory, sleep, and critical care medicine journals to consolidate contemporary best practices and recommendations related to prediction study design, conduct, and reporting. Herein, we address issues commonly encountered in submissions to our various journals. Key topics include considerations for selecting predictor variables, operationalizing variables, dealing with missing data, the importance of appropriate validation, model performance measures and their interpretation, and good reporting practices. Supplemental discussion covers emerging topics such as model fairness, competing risks, pitfalls of “modifiable risk factors”, measurement error, and risk for bias. This guidance is not meant to be overly prescriptive; we acknowledge that every study is different, and no set of rules will fit all cases. Additional best practices can be found in the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines, to which we refer readers for further details.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/ccm.0000000000004246

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Many ICU studies involving AI methods still fail to meet these standards, leading to concerns about inadequate external validation and generalizability.

    “Despite the availability of these frameworks, many ICU studies involving AI methods still fail to meet these standards, leading to concerns about inadequate external validation and generalizability [68, 72, 73].”

    From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [73]

    The Model Validation section (page unavailable) says validation is necessary and that models can fail to generalize to new data, supporting the concern about validation and generalizability. It does not establish that many ICU studies using AI fail to meet the guidance or lack external validation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. [1]Altman DG Vergouwe Y Royston P Prognosis and prognostic research: Validating a prognostic model. BMJ 2009; 338: b605 19477892
  2. [2]Moons KG Altman DG Vergouwe Y Prognosis and prognostic research: Application and impact of prognostic models in clinical practice. BMJ 2009; 338: b606 19502216
  3. [3]Moons KG Royston P Vergouwe Y Prognosis and prognostic research: What, why, and how? BMJ 2009; 338: b375 19237405
  4. [4]Royston P Moons KG Altman DG Prognosis and prognostic research: Developing a prognostic model. BMJ 2009; 338: b604 19336487
  5. [5]Steyerberg E Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2019 Second Edition New York, Springer International Publishing.
  6. [6]Collins GS Reitsma JB Altman DG Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med 2015; 162: 735– 736 25984857
  7. [9]Shmueli G To explain or to predict? Statist Sci 2010; 25: 289– 310
  8. [10]Zimmerman JE Kramer AA McNair DS Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006; 34: 1297– 1310 16540951
  9. [12]Braun L Race, ethnicity and lung function: A brief history. Can J Respir Ther 2015; 51: 99– 101 26566381 PMC4631137
  10. [15]Rajkomar A Hardt M Howell MD Ensuring fairness in machine learning to advance health equity. Ann Intern Med 2018; 169: 866– 872 30508424 PMC6594166
  11. [17]Smith G Step away from stepwise. J Big Data 2018; 5: 32
  12. [19]Mundry R Nunn CL Stepwise model fitting and statistical inference: Turning noise into signal pollution. Am Nat 2009; 173: 119– 123 19049440
  13. [20]Walter S Tiemeier H Variable selection: current practice in epidemiological studies. Eur J Epidemiol 2009; 24: 733– 736 19967429 PMC2791468
  14. [23]Riley RD Snell KIE Ensor J Minimum sample size for developing a multivariable prediction model: Part I - Continuous outcomes. Stat Med 2019; 38: 1262– 1275 30347470
  15. [26]van Smeden M Moons KG de Groot JA Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat Methods Med Res. 2019; 28: 2455– 2474 29966490 PMC6710621
  16. [27]Senn S Dichotomania: An obsessive compulsive disorder that is badly affecting the quality of analysis of pharmaceutical trials. In: Proceedings of the International Statistical Institute, 55th Session, 2005, April 6-12, 2005 Sydney, Australia
  17. [28]Royston P Altman DG Sauerbrei W Dichotomizing continuous predictors in multiple regression: A bad idea. Stat Med 2006; 25: 127– 141 16217841
  18. [29]Sharafoddini A Dubin JA Maslove DM A new insight into missing data in intensive care unit patient profiles: Observational study. JMIR Med Inform 2019; 7: e11605 30622091 PMC6329436
  19. [30]Ware JH Harrington D Hunter DJ Missing data. N Engl J Med 2012; 367: 1353– 1354
  20. [31]Newgard CD Haukoos JS Advanced statistics: Missing data in clinical research–Part 2: Multiple imputation. Acad Emerg Med 2007; 14: 669– 678 17595237
  21. [32]Altman DG Royston P What do we mean by validating a prognostic model? Stat Med 2000; 19: 453– 473 10694730
  22. [33]Steyerberg EW Vergouwe Y Towards better clinical prediction models: Seven steps for development and an ABCD for validation. Eur Heart J 2014; 35: 1925– 1931 24898551 PMC4155437
  23. [35]Debray TP Vergouwe Y Koffijberg H A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279– 289 25179855
  24. [36]Vergouwe Y Moons KG Steyerberg EW External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971– 980 20807737 PMC2984249
  25. [37]Steyerberg EW Harrell FE Jr Borsboom GJ Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001; 54: 774– 781 11470385
  26. [38]Lee J Maslove DM Customization of a severity of illness score using local electronic medical record data. J Intensive Care Med 2017; 32: 38– 47 25969432
  27. [39]Steyerberg EW Vickers AJ Cook NR Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology 2010; 21: 128– 138 20010215 PMC3575184
  28. [40]Alba AC Agoritsas T Walsh M Discrimination and calibration of clinical prediction models: Users’ guides to the medical literature. JAMA 2017; 318: 1377– 1384 29049590
  29. [41]Leisman DE Rare events in the ICU: An emerging challenge in classification and prediction. Crit Care Med 2018; 46: 418– 424 29474323

Cited by

Sorted by citations

Authors
Leisman, D. E., Harhay, M. O., Lederer, D. J., Abramson, M., Adjei, A. A., Bakker, J., Ballas, Z. K., Barreiro, E., Bell, S. C., Bellomo, R., Bernstein, J. A., Branson, R. D., Brusasco, V., Chalmers, J. D., Chokroverty, S., Citerio, G., Collop, N. A., Cooke, C. R., Crapo, J. D., Donaldson, G., Fitzgerald, D. A., Grainger, E., Hale, L., Herth, F. J., Kochanek, P. M., Marks, G., Moorman, J. R., Ost, D. E., Schatz, M., Sheikh, A., Smyth, A. R., Stewart, I., Stewart, P. W., Swenson, E. R., Szymusiak, R., Teboul, J.-L., Vincent, J.-L., Wedzicha, J. A., & Maslove, D. M.
Affiliations
  • Icahn School of Medicine at Mount Sinai (US)
  • University of Pennsylvania (US)
  • Regeneron (United States) (US)
  • Monash University (AU)
  • Mayo Clinic (US)
  • New York University (US)
  • University of Iowa (US)
  • Hospital Del Mar (ES)
  • Prince Charles Hospital (AU)
  • The University of Melbourne (AU)
  • University of Cincinnati (US)
  • University of Genoa (IT)
  • University of Dundee (GB)
  • Hackensack Meridian Health (US)
  • University of Milano-Bicocca (IT)
  • Emory University (US)
  • University of Michigan (US)
  • National Jewish Health (US)
  • Imperial College London (GB)
  • The University of Sydney (AU)
  • The Lancet, London, United Kingdom.
  • Stony Brook Medicine (US)
  • Heidelberg University (DE)
  • University of Pittsburgh (US)
  • University of Virginia (US)
  • The University of Texas MD Anderson Cancer Center (US)
  • Kaiser Permanente (US)
  • University of Edinburgh (GB)
  • University of Nottingham (GB)
  • University of North Carolina at Chapel Hill (US)
  • University of Washington (US)
  • University of California, Los Angeles (US)
  • Bicêtre Hospital (FR)
  • Université Libre de Bruxelles (BE)
  • Queen's University (CA)
Corresponding author
Maslove, David M.
ORCID
Venue
Critical Care Medicine
Publisher
Ovid Technologies (Wolters Kluwer Health)
Published
2020-05
Volume
48
Issue
5
Pages
623-633
ISSN
0090-3493 (ISSN-L)
DOI
10.1097/ccm.0000000000004246
PMC
PMC7161722
OpenAlex
W3010095828
Cited by
271
References
12
Citations by year
2026: 20 · 2025: 28 · 2024: 26 · 2023: 40 · 2022: 71 · 2021: 53 · 2020: 31 · 2012: 1
FWCI
34.38
Type
journal article
Language
en
Open access
Licence
Locations
  • Critical Care Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • Monash University Research Portal (Monash University) (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Discovery Research Portal (University of Dundee) (publishedVersion, cc-by-nc-nd)
  • Repositori digital de la UPF (Universitat Pompeu Fabra) (publishedVersion, cc-by-nc-nd)
  • Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by-nc-nd)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by-nc-nd)
  • HAL (submittedVersion)
  • BOA (University of Milano-Bicocca) (submittedVersion)
  • Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) (submittedVersion, other-oa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
  • Spiral (Imperial College London) (submittedVersion, cc-by-nc-nd)
Funders
  • U.S. Department of Veterans Affairs (IK6 BX004208)
  • National Institute for Health and Care Research (NF-SI-0510-10270, NF-SI-0513-10077)
  • National Heart, Lung, and Blood Institute (R00HL141678, K99HL141678)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
prediction model, model validation, model updating, predictor selection, missing data, model performance measures, model fairness, competing risks, modifiable risk factors, measurement error, risk of bias, TRIPOD guidelines, respiratory medicine, sleep medicine
MeSH terms
Critical Care, Decision Support Techniques, Humans, Periodicals as Topic, Prognosis, Respiratory Tract Diseases, Sleep Wake Disorders, Reproducibility of Results, Models, Statistical, Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review