Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study
Open accessAbstract
Background: In clinical prediction modelling, missing data can occur at any stage of the model pipeline; development, validation or deployment. Multiple imputation is often recommended yet challenging to apply at deployment; for example, the outcome cannot be in the imputation model, as recommended under multiple imputation. Regression imputation uses a fitted model to impute the predicted value of missing predictors from observed data, and could offer a pragmatic alternative at deployment. Moreover, the use of missing indicators has been proposed to handle informative missingness, but it is currently unknown how well this method performs in the context of clinical prediction models. Methods: We simulated data under various missing data mechanisms to compare the predictive performance of clinical prediction models developed using both imputation methods. We consider deployment scenarios where missing data is permitted or prohibited, imputation models that use or omit the outcome, and clinical prediction models that include or omit missing indicators. We assume that the missingness mechanism remains constant across the model pipeline. We also apply the proposed strategies to critical care data. Results: With complete data available at deployment, our findings were in line with existing recommendations; that the outcome should be used to impute development data when using multiple imputation and omitted under regression imputation. When missingness is allowed at deployment, omitting the outcome from the imputation model at the development was preferred. Missing indicators improved model performance in many cases but can be harmful under outcome-dependent missingness. Conclusion: We provide evidence that commonly taught principles of handling missing data via multiple imputation may not apply to clinical prediction models, particularly when data can be missing at deployment. We observed comparable predictive performance under multiple imputation and regression imputation. The performance of the missing data handling method must be evaluated on a study-by-study basis, and the most appropriate strategy for handling missing data at development should consider whether missing data are allowed at deployment. Some guidance is provided.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Single imputation can perform well, although multiple imputation tends to be more consistent and stable.
“Sisk and colleagues showed that single imputation can perform well, although multiple imputation tends to be more consistent and stable.59”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
<b>mice</b>: Multivariate Imputation by Chained Equations in<i>R</i>
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · see citing works
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · see citing works
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299 · see citing works
Seymour, C. W., Liu, V. X., Iwashyna, T. J., Brunkhorst, F. M., Rea, T. D., Scherag, A., Rubenfeld, G., Kahn, J. M., Shankar-Hari, M., Singer, M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0288 · Cited by 3,939 · see citing works
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Antlanger, M., Aschauer, S., Kopecky, C., Hecking, M., Kovarik, J. J., Werzowa, J. M., Mascherbauer, J., Genser, B., Säemann, M. D., & Bonderman, D. (2017) · Kidney & Blood Pressure Research · doi:10.1159/000473868 · Cited by 2,808 · see citing works
Jakobsen, J. C., Gluud, C., Wetterslev, J., & Winkel, P. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0442-1 · Cited by 2,747 · see citing works
Multiple Organ Dysfunction Score
Marshall, J. C., Cook, D. J., Christou, N. V., Bernard, G. R., Sprung, C. L., & Sibbald, W. J. (1995) · Critical Care Medicine · doi:10.1097/00003246-199510000-00007 · Cited by 2,531 · see citing works
Flexible Imputation of Missing Data, Second Edition
van Buuren, S. (2018) · doi:10.1201/9780429492259 · Cited by 2,273 · see citing works
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928 · see citing works
Hippisley-Cox, J., Coupland, C., & Brindle, P. (2017) · BMJ · doi:10.1136/bmj.j2099 · Cited by 1,703 · see citing works
Steyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Moreno, R. P., Metnitz, P., Almeida, E., Jordan, B., Bauer, P., Campos, R. A., Iapichino, G., Edbrooke, D. L., Capuzzo, M., Gall, J. L., et al. (2005) · Intensive Care Medicine · doi:10.1007/s00134-005-2763-5 · Cited by 1,414 · see citing works
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
Using simulation studies to evaluate statistical methods
Morris, T. P., White, I. R., & Crowther, M. J. (2019) · Statistics in Medicine · doi:10.1002/sim.8086 · Cited by 1,354 · see citing works
Using simulation studies to evaluate statistical methods
Morris, T., White, I. R., & Crowther, M. (2019) · UCL Discovery (University College London) · Cited by 1,011 · see citing works
Using the outcome for imputation of missing predictor values was preferred
Moons, K. G. M., Donders, R. A. R. T., Stijnen, T., & Harrell, F. E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.009 · Cited by 994 · see citing works
Marshall, A., Altman, D. G., HOLDER, R. L., & Royston, P. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-57 · Cited by 883 · see citing works
Accounting for missing data in statistical analyses: multiple imputation is not always the answer
Hughes, R. A., Heron, J., Sterne, J. A. C., & Tilling, K. (2019) · International Journal of Epidemiology · doi:10.1093/ije/dyz032 · Cited by 802 · see citing works
Sample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K., de Groot, J. D., Collins, G., Altman, D., Eijkemans, M., & Reitsma, J. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 776 · see citing works
Multiple Imputation for Interval Estimation From Simple Random Samples With Ignorable Nonresponse
Rubin, D. B., & Nathaniel Schenker, A. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478280 · Cited by 768 · see citing works
Groenwold, R. H. H., White, I. R., Donders, A. R. T., Carpenter, J. R., Altman, D. G., & Moons, K. G. M. (2012) · Canadian Medical Association Journal · doi:10.1503/cmaj.110977 · Cited by 538 · see citing works
Clinical prediction models: diagnosis versus prognosis
van Smeden, M., Reitsma, J. B., Riley, R. D., Collins, P. G. S., & Moons, K. G. M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.009 · Cited by 241 · see citing works
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211 · see citing works
Hippisley‐Cox, J., & Coupland, C. A. C. (2010) · BMC Family Practice · doi:10.1186/1471-2296-11-49 · Cited by 178 · see citing works
Dealing with Missing Predictor Values When Applying Clinical Prediction Models
Janssen, K. J. M., Vergouwe, Y., Donders, A. R. T., Harrell, F. E., Chen, Q., Grobbee, D. E., & Moons, K. G. M. (2009) · Clinical Chemistry · doi:10.1373/clinchem.2008.115345 · Cited by 152 · see citing works
Haneuse, S., & Daniels, M. J. (2016) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1203 · Cited by 118 · see citing works
Missing data should be handled differently for prediction than for description or causal explanation
Sperrin, M., Martin, G. P., Sisk, R., & Peek, N. B. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.028 · Cited by 112 · see citing works
Self-citationImputation is beneficial for handling missing data in predictive models
Steyerberg, E. W., & van Veen, M. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.003 · Cited by 109 · see citing works
Sick patients have more data: the non-random completeness of electronic health records.
Weiskopf, N. G., Rusanov, A., & Weng, C. (2013) · Europe PMC (PubMed Central) · Cited by 99 · see citing works
Wood, A. M., Royston, P., & White, I. R. (2015) · Biometrical Journal · doi:10.1002/bimj.201400004 · Cited by 78 · see citing works
Missing data was handled inconsistently in UK prediction models: a review of method used
Tsvetanova, A., Sperrin, M., Peek, N. B., Buchan, I. E., Hyland, S. L., & Martin, G. P. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.09.008 · Cited by 63 · see citing works
Self-citationMissing data and prediction: the pattern submodel
Mercaldo, S., & Blume, J. D. (2018) · Biostatistics · doi:10.1093/biostatistics/kxy040 · Cited by 61 · see citing works
Sisk, R., Lijing Lin, A., Sperrin, M., Barrett, J., Tom, B. D. M., Diaz‐Ordaz, K., Peek, N. B., & Martin, G. P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa242 · Cited by 46 · see citing works
Self-citationNijman, S. W. J., Groenhof, T. K. J., Hoogland, J., Bots, M. L., Brandjes, M., Jacobs, J. J. L., Asselbergs, F. W., Moons, K. G. M., & Debray, T. P. A. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.003 · Cited by 46 · see citing works
Multiple imputation with missing indicators as proxies for unmeasured variables: simulation study
Sperrin, M., & Martin, G. P. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01068-x · Cited by 41 · see citing works
Self-citationPate, A., Emsley, R., Sperrin, M., Martin, G. P., & van Staa, T. P. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00082-3 · Cited by 36 · see citing works
Self-citationvan Smeden, M., Groenwold, R. H. H., & Moons, K. G. M. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.007 · Cited by 33 · see citing works
On the Relation between Prediction and Imputation Accuracy under Missing Covariates
Ramosaj, B., Tulowietzki, J., & Pauly, M. (2022) · Entropy · doi:10.3390/e24030386 · Cited by 13 · see citing works
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Flagged first, then by citations
Cited by
Sorted by citations
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Barreñada, L., Ledger, A., Dhiman, P., Collins, G., Wynants, L., Verbakel, J. Y., Timmerman, D., Valentin, L., & Van Calster, B. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000817 · Cited by 42 · see citing works
Clinical prediction models using machine learning in oncology: challenges and recommendations
Collins, G. S., Chester-Jones, M., Gerry, S., Ma, J., Matos, J., Sehjal, J., Tsegaye, B., & Dhiman, P. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000914 · Cited by 27 · see citing works
- Authors
- Sisk, R., Sperrin, M., Peek, N., van Smeden, M., & Martin, G. P.
- Affiliations
- Manchester Academic Health Science Centre (GB)
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
- Corresponding author
- Sisk, Rose
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2023-04-27
- Volume
- 32
- Issue
- 8
- Pages
- 1461-1477
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- PMC
- PMC10515473
- OpenAlex
- W4367172397
- Cited by
- 99
- References
- 42
- Citations by year
- 2026: 47 · 2025: 31 · 2024: 18 · 2023: 3
- FWCI
- 20.13
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Statistical Methods in Medical Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, cc-by-nc)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- Alan Turing Institute
- Medical Research Council (MR/N013751/11)
- Gendius Ltd
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- missing data, clinical prediction model, multiple imputation, regression imputation, informative missingness, model deployment, predictive performance, missing data mechanisms
- MeSH terms
- Computer Simulation, Critical Care, Data Interpretation, Statistical, Humans, Research Design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
