Outcome-sensitive multiple imputation: a simulation study
Open accessAbstract
BACKGROUND: Multiple imputation is frequently used to deal with missing data in healthcare research. Although it is known that the outcome should be included in the imputation model when imputing missing covariate values, it is not known whether it should be imputed. Similarly no clear recommendations exist on: the utility of incorporating a secondary outcome, if available, in the imputation model; the level of protection offered when data are missing not-at-random; the implications of the dataset size and missingness levels. METHODS: We used realistic assumptions to generate thousands of datasets across a broad spectrum of contexts: three mechanisms of missingness (completely at random; at random; not at random); varying extents of missingness (20-80% missing data); and different sample sizes (1,000 or 10,000 cases). For each context we quantified the performance of a complete case analysis and seven multiple imputation methods which deleted cases with missing outcome before imputation, after imputation or not at all; included or did not include the outcome in the imputation models; and included or did not include a secondary outcome in the imputation models. Methods were compared on mean absolute error, bias, coverage and power over 1,000 datasets for each scenario. RESULTS: Overall, there was very little to separate multiple imputation methods which included the outcome in the imputation model. Even when missingness was quite extensive, all multiple imputation approaches performed well. Incorporating a secondary outcome, moderately correlated with the outcome of interest, made very little difference. The dataset size and the extent of missingness affected performance, as expected. Multiple imputation methods protected less well against missingness not at random, but did offer some protection. CONCLUSIONS: As long as the outcome is included in the imputation model, there are very small performance differences between the possible multiple imputation approaches: no outcome imputation, imputation or imputation and deletion. All informative covariates, even with very high levels of missingness, should be included in the multiple imputation model. Multiple imputation offers some protection against a simple missing not at random mechanism.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
If no auxiliary variables were used in the imputations, people with imputed outcomes can be removed, and the model can be developed based only on people with observed outcomes.
“If no auxiliary variables were used in the imputations, people with imputed outcomes can be removed, and the model can be developed based only on people with observed outcomes.57”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701 · see citing works
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638 · see citing works
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299 · see citing works
Self-citationMultiple imputation after 18+ years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476908 · Cited by 4,072 · see citing works
Multiple Imputation After 18+ Years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.2307/2291635 · Cited by 2,686 · see citing works
4. Regression with Missing Ys: An Improved Strategy for Analyzing Multiply Imputed Data
von Hippel, P. T. (2007) · Sociological Methodology · doi:10.1111/j.1467-9531.2007.00180.x · Cited by 1,495 · see citing works
Using the outcome for imputation of missing predictor values was preferred
Moons, K. G. M., Donders, R. A. R. T., Stijnen, T., & Harrell, F. E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.009 · Cited by 994 · see citing works
White, I. R., & Carlin, J. B. (2010) · Statistics in Medicine · doi:10.1002/sim.3944 · Cited by 751 · see citing works
Self-citationMissing covariate data in medical research: To impute is better than to ignore
Janssen, K. J. M., Donders, A. R. T., Harrell, F. E., Vergouwe, Y., Chen, Q., Grobbee, D. E., & Moons, K. G. M. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.12.008 · Cited by 593 · see citing works
Dealing With Missing Outcome Data in Randomized Trials and Observational Studies
Groenwold, R. H. H., Donders, A. R. T., Roes, K. C. B., Harrell, F. E., & Moons, K. G. M. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kwr302 · Cited by 398 · see citing works
Handling missing data in RCTs; a review of the top medical journals
Bell, M. L., Fiero, M., Horton, N. J., & Hsu, C.-H. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-118 · Cited by 395 · see citing works
Inference for imputation estimators
Robins, J. M. (2000) · Biometrika · doi:10.1093/biomet/87.1.113 · Cited by 278 · see citing works
Kontopantelis, E., & Reeves, D. (2010) · Statistical Methods in Medical Research · doi:10.1177/0962280210392008 · Cited by 264 · see citing works
Self-citationEmployee attitude surveys: Examining the attitudes of noncompliant employees.
Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.2.284 · Cited by 139 · see citing works
The Incredible Years Teacher Classroom Management Program: Outcomes from a Group Randomized Trial
Reinke, W. M., Herman, K. C., & Dong, N. (2018) · Prevention Science · doi:10.1007/s11121-018-0932-3 · Cited by 88 · see citing works
Sullivan, T. R., Salter, A. B., Ryan, P., & Lee, K. J. (2015) · American Journal of Epidemiology · doi:10.1093/aje/kwv100 · Cited by 86 · see citing works
Eliciting and using expert opinions about dropout bias in randomized controlled trials
White, I. R., Carpenter, J. R., Evans, S. J. W., & Schroter, S. (2007) · Clinical Trials · doi:10.1177/1740774507077849 · Cited by 85 · see citing works
Self-citationChanges in Arterial Access Site and Association With Mortality in the United Kingdom
Mamas, M. A., Nolan, J., de Belder, M., Zaman, A., Kinnaird, T. D., Curzen, N. P., Kwok, C. S., Buchan, I. E., Ludman, P., & Kontopantelis, E. (2016) · Circulation · doi:10.1161/circulationaha.115.018083 · Cited by 76 · see citing works
Self-citationMarshall, M. B., Peat, G., Nicholls, E. E., Myers, H. L., Mamas, M. A., & van der Windt, D. (2018) · Scandinavian Journal of Rheumatology · doi:10.1080/03009742.2018.1459831 · Cited by 35 · see citing works
Simulation-Based Power Calculations for Mixed Effects Modeling: ipdpower in Stata
Kontopantelis, E., Springate, D. A., Parisi, R., & Reeves, D. (2016) · Journal of Statistical Software · doi:10.18637/jss.v074.i12 · Cited by 33 · see citing works
Self-citationLu, K., Jiang, L., & Tsiatis, A. A. (2010) · Biometrics · doi:10.1111/j.1541-0420.2010.01405.x · Cited by 21 · see citing works
Bieleman, A., Stewart, R. E., Reneman, M. F., van Ittersum, W. M., van der Schans, C. P., Drossaers-Bakker, K. W., & Oosterveld, F. G. J. (2018) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-018-9809-3 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
- Authors
- Kontopantelis, E., White, I. R., Sperrin, M., & Buchan, I.
- Affiliations
- University of Manchester (GB)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Kontopantelis, Evangelos
- Venue
- BMC Medical Research Methodology
- Published
- 2017-01-09
- Volume
- 17
- Issue
- 1
- Pages
- 2-2
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC5220613
- OpenAlex
- W2569711455
- Cited by
- 205
- References
- 22
- Citations by year
- 2026: 32 · 2025: 23 · 2024: 26 · 2023: 20 · 2022: 34 · 2021: 27 · 2020: 19 · 2019: 6 · 2018: 12 · 2017: 5 · 2016: 1
- FWCI
- 7.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- Apollo
- Funders
- Medical Research Council (MR/K006665/1)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- multiple imputation, missing data, missing not at random, complete case analysis, secondary outcomes, missing data mechanisms, sample size
- MeSH terms
- Computer Simulation, Data Interpretation, Statistical, Humans, Research Design, Models, Statistical, Observational Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
