Bias and Precision of the “Multiple Imputation, Then Deletion” Method for Dealing With Missing Outcome Data
Abstract
Multiple imputation (MI) is increasingly being used to handle missing data in epidemiologic research. When data on both the exposure and the outcome are missing, an alternative to standard MI is the "multiple imputation, then deletion" (MID) method, which involves deleting imputed outcomes prior to analysis. While MID has been shown to provide efficiency gains over standard MI when analysis and imputation models are the same, the performance of MID in the presence of auxiliary variables for the incomplete outcome is not well understood. Using simulated data, we evaluated the performance of standard MI and MID in regression settings where data were missing on both the outcome and the exposure and where an auxiliary variable associated with the incomplete outcome was included in the imputation model. When the auxiliary variable was unrelated to missingness in the outcome, both standard MI and MID produced negligible bias when estimating regression parameters, with standard MI being more efficient in most settings. However, when the auxiliary variable was also associated with missingness in the outcome, alarmingly MID produced markedly biased parameter estimates. On the basis of these results, we recommend that researchers use standard MI rather than MID in the presence of auxiliary variables associated with an incomplete outcome.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
If imputation incorporates auxiliary variables, including those with imputed outcomes in the model development is advisable.
“However, if imputation incorporates auxiliary variables, including those with imputed outcomes in the model development is advisable.58”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701 · see citing works
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,384 · see citing works
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638 · see citing works
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925 · see citing works
Multiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · see citing works
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299 · see citing works
Analysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012 · see citing works
Analysis of Incomplete Multivariate Data
Booth, D. E., & Schafer, J. L. (2000) · Technometrics · doi:10.2307/1271466 · Cited by 5,627 · see citing works
Statistical Analysis With Missing Data
Ghosh, S., Little, R. J. A., & Rubin, D. B. (1988) · Technometrics · doi:10.2307/1269814 · Cited by 4,730 · see citing works
Schafer, J. L. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800102 · Cited by 3,949 · see citing works
How Many Imputations are Really Needed? Some Practical Clarifications of Multiple Imputation Theory
Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007) · Prevention Science · doi:10.1007/s11121-007-0070-9 · Cited by 2,795 · see citing works
A comparison of inclusive and restrictive strategies in modern missing data procedures.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.330 · Cited by 2,389 · see citing works
Allison, P. (2002) · doi:10.4135/9781412985079 · Cited by 2,050 · see citing works
The prevention and treatment of missing data in clinical trials
Little, R. J., D'Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Frangakis, C., Hogan, J. W., Molenberghs, G., Murphy, S. A., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmsr1203730 · Cited by 1,884 · see citing works
Multiple Imputation for Multivariate Missing-Data Problems: A Data Analyst's Perspective
Schafer, J. L., & Olsen, M. K. (1998) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3304_5 · Cited by 1,518 · see citing works
4. Regression with Missing Ys: An Improved Strategy for Analyzing Multiply Imputed Data
von Hippel, P. T. (2007) · Sociological Methodology · doi:10.1111/j.1467-9531.2007.00180.x · Cited by 1,495 · see citing works
Fully conditional specification in multivariate imputation
Van Buuren, S., Brand, J. P. L., Groothuis-Oudshoorn, C. G. M., & Rubin, D. B. (2006) · Journal of Statistical Computation and Simulation · doi:10.1080/10629360600810434 · Cited by 1,217 · see citing works
Regression with Missing X 's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10476282 · Cited by 968 · see citing works
Graham, J. A. (2012) · Springer eBooks · doi:10.1007/978-1-4614-4018-5 · Cited by 799 · see citing works
Lee, K. J., & Carlin, J. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwp425 · Cited by 787 · see citing works
Self-citationWhite, I. R., & Carlin, J. (2010) · Statistics in Medicine · doi:10.1002/sim.3944 · Cited by 781 · see citing works
Regression With Missing X's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.2307/2290664 · Cited by 647 · see citing works
Multiple imputation: review of theory, implementation and software
Harel, O., & Zhou, X. (2007) · Statistics in Medicine · doi:10.1002/sim.2787 · Cited by 427 · see citing works
Robustness of a multivariate normal approximation for imputation of incomplete binary data
Bernaards, C. A., Belin, T. R., & Schafer, J. L. (2006) · Statistics in Medicine · doi:10.1002/sim.2619 · Cited by 191 · see citing works
Longitudinal Associations Between Poverty and Obesity From Birth Through Adolescence
Lee, H. E., Andrew, M., Gebremariam, A., Lumeng, J. C., & Lee, J. M. (2014) · American Journal of Public Health · doi:10.2105/ajph.2013.301806 · Cited by 122 · see citing works
Arterial Stiffness, Physical Function, and Functional Limitation
Brunner, E. J., Shipley, M. J., Witte, D. R., Singh‐Manoux, A., Britton, A. R., Tabák, Á. G., McEniery, C. M., Wilkinson, I. B., & Mika Kivimaki, B. (2011) · Hypertension · doi:10.1161/hypertensionaha.110.168864 · Cited by 106 · see citing works
Recovery of information from multiple imputation: a simulation study
Lee, K. J., & Carlin, J. B. (2012) · Emerging Themes in Epidemiology · doi:10.1186/1742-7622-9-3 · Cited by 105 · see citing works
Self-citationGroen, E. H., Lingsma, H. F., Caudri, D., Levie, D., Wijga, A. H., Koppelman, G. H., Duijts, L., Jaddoe, V. W. V., Smit, H. A., Kerkhof, M., et al. (2013) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2013.07.007 · Cited by 65 · see citing works
Zuidersma, M., Conradi, H. J., van Melle, J. P., Ormel, J., & de Jonge, P. (2012) · International Journal of Cardiology · doi:10.1016/j.ijcard.2012.07.002 · Cited by 60 · see citing works
Effects of Physical Activity on Teen Smoking Cessation
Horn, K. A., Dino, G., Branstetter, S. A., Zhang, J., Noerachmanto, N., Jarrett, T. D., & Taylor, M. (2011) · PEDIATRICS · doi:10.1542/peds.2010-2599 · Cited by 58 · see citing works
Association of Coexisting Diabetes and Depression With Mortality After Myocardial Infarction
Bot, M., Pouwer, F., Zuidersma, M., van Melle, J. P., & de Jonge, P. (2012) · Diabetes Care · doi:10.2337/dc11-1749 · Cited by 48 · see citing works
Hiyoshi, A., Fukuda, Y., Shipley, M. J., & Brunner, E. J. (2013) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2013-202608 · Cited by 27 · see citing works
The Effects of Auxiliary Variables on Coefficient Bias and Efficiency in Multiple Imputation
Mustillo, S. (2012) · Sociological Methods & Research · doi:10.1177/0049124112452392 · Cited by 21 · see citing works
Anti-Gay Prejudice and All-Cause Mortality Among Heterosexuals in the United States
Hatzenbuehler, M. L., Bellatorre, A. C., & Muennig, P. A. (2013) · American Journal of Public Health · doi:10.2105/ajph.2013.301678 · Cited by 14 · see citing works
Multiple imputation of missing blood pressure covariates in survival analysis
van Buuren, S., Boshuizen, H. C., & Knook, D. L. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990330)18:6
Flagged first, then by citations
Research works printed but not linked: 1
- Sullivan et al. Am J Epidemiol. 2015;182(6):528-534
Cited by
Sorted by citations
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Outcome-sensitive multiple imputation: a simulation study
Kontopantelis, E., White, I. R., Sperrin, M., & Buchan, I. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0281-5 · Cited by 205 · see citing works
Eriksson, L. S. E., Epstein, E., Testa, A. C., Fischerova, D., Valentin, L., Sladkevicius, P., Franchi, D., Frühauf, F., Fruscio, R., Haak, L. A., et al. (2020) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.21950 · Cited by 24 · see citing works
- Authors
- Sullivan, T. R., Salter, A. B., Ryan, P., & Lee, K. J.
- Affiliations
- The University of Adelaide (AU)
- Corresponding author
- Sullivan, Thomas R.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-09-02
- Volume
- 182
- Issue
- 6
- Pages
- 528-534
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2136953071
- Cited by
- 86
- References
- 35
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 6 · 2023: 8 · 2022: 8 · 2021: 10 · 2020: 10 · 2019: 9 · 2018: 7 · 2017: 11 · 2016: 5 · 2015: 1
- FWCI
- 7.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- Funders
- Australian Postgraduate Award
- National Health and Medical Research Council Career Development Fellowship (1053609)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- multiple imputation, missing outcome data, missing data, auxiliary variables, bias, efficiency, regression coefficient estimation, imputation models
- MeSH terms
- Computer Simulation, Data Interpretation, Statistical, Humans, Monte Carlo Method, Cluster Analysis, Randomized Controlled Trials as Topic, Treatment Outcome, Propensity Score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
