The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Should multiple imputation be the method of choice for handling missing data in randomized trials?
Open accessAbstract
The use of multiple imputation has increased markedly in recent years, and journal reviewers may expect to see multiple imputation used to handle missing data. However in randomized trials, where treatment group is always observed and independent of baseline covariates, other approaches may be preferable. Using data simulation we evaluated multiple imputation, performed both overall and separately by randomized group, across a range of commonly encountered scenarios. We considered both missing outcome and missing baseline data, with missing outcome data induced under missing at random mechanisms. Provided the analysis model was correctly specified, multiple imputation produced unbiased treatment effect estimates, but alternative unbiased approaches were often more efficient. When the analysis model overlooked an interaction effect involving randomized group, multiple imputation produced biased estimates of the average treatment effect when applied to missing outcome data, unless imputation was performed separately by randomized group. Based on these results, we conclude that multiple imputation should not be seen as the only acceptable way to handle missing data in randomized trials. In settings where multiple imputation is adopted, we recommend that imputation is carried out separately by randomized group.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638
Multiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · cited 1×
“For MI, linear and logistic regression were used for the imputation of continuous and binary variables, respectively, with m1⁄4 50 imputations based on the rule of thumb that the number of imputations should at least equal the percentage of missing data.(17) Completed datasets were analysed using linear and logistic r…”
MethodSelf-citationRandom-Effects Models for Longitudinal Data
Laird, N. M., & Ware, J. H. (1982) · Biometrics · doi:10.2307/2529876 · Cited by 8,923
Generalized linear mixed models: a practical guide for ecology and evolution
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J.-S. S. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2008.10.008 · Cited by 8,910 · cited 1×
“Generalized mixed models can be implemented in a similar manner to LMMs for continuous data if subject-specific estimates are of interest(9); however, these models can be challenging to fit given the variety of estimation procedures available and the computational difficulties that can arise with large numbers of repe…”
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299
Self-citationMissing data analysis: making it work in the real world
Graham, J. W. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085530 · Cited by 6,163
Analysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · cited 1×
“Treatment effect estimates for LMMs in this article were obtained using restricted maximum likelihood estimation with degrees of freedom calculated according to the Kenward–Roger method.(47) MI overall, MI by group and the LMM produced unbiased treatment effect estimates across the 16 simulation scenarios (4 missing d…”
MethodModels for Longitudinal Data: A Generalized Estimating Equation Approach
Zeger, S. L., Liang, K.-Y., & Albert, P. S. (1988) · Biometrics · doi:10.2307/2531734 · Cited by 4,505
Multiple imputation of discrete and continuous data by fully conditional specification
van Buuren, S. (2007) · Statistical Methods in Medical Research · doi:10.1177/0962280206074463 · Cited by 2,827
Review: a gentle introduction to imputation of missing values.
Donders, A. R. T., van der Heijden, G. J. M. G., Stijnen, T., & Moons, K. G. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.014 · Cited by 2,582
A comparison of inclusive and restrictive strategies in modern missing data procedures.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.330 · Cited by 2,389
The prevention and treatment of missing data in clinical trials
Little, R. J., D'Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Frangakis, C., Hogan, J. W., Molenberghs, G., Murphy, S. A., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmsr1203730 · Cited by 1,884 · cited 1×
“By exploiting information in partially observed cases, MI and likelihood-based approaches have been favoured over CCA for the analysis of multivariate outcomes.(9,11,39,40) In what follows, we briefly introduce likelihood approaches for multivariate outcome data, describe the link between intermediate outcome measures…”
MethodReview of inverse probability weighting for dealing with missing data
Seaman, S. R., & White, I. R. (2013) · Statistical Methods in Medical Research · doi:10.1177/0962280210395740 · Cited by 1,654
Self-citation4. Regression with Missing Ys: An Improved Strategy for Analyzing Multiply Imputed Data
von Hippel, P. T. (2007) · Sociological Methodology · doi:10.1111/j.1467-9531.2007.00180.x · Cited by 1,495
Regression with Missing X 's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10476282 · Cited by 968
Multiple Imputation and its Application
Carpenter, J. R., & Kenward, M. G. (2013) · doi:10.1002/9781119942283 · Cited by 913
Makrides, M., Gibson, R. A., McPhee, A. J., Yelland, L., Quinlivan, J., Ryan, P., & DOMInO Investigative Team, A. T. (2010) · JAMA · doi:10.1001/jama.2010.1507 · Cited by 559
Self-citationGroenwold, R. H. H., White, I. R., Donders, A. R. T., Carpenter, J. R., Altman, D. G., & Moons, K. G. M. (2012) · Canadian Medical Association Journal · doi:10.1503/cmaj.110977 · Cited by 538 · cited 1×
“Despite being inappropriate for general use,(49,50) the missing indicator method has been validated for addressing missing covariate data in RCTs, where X and T are independent and missingness in X is conditionally independent of Y.(48,51) For strong correlations between X and Y, White and Thompson(48) found that mean…”
MethodSelf-citationSnowden, J. M., Rose, S., & Mortimer, K. M. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwq472 · Cited by 511
Kahan, B. C., Jairath, V., Doré, C. J., & Morris, T. P. (2014) · Trials · doi:10.1186/1745-6215-15-139 · Cited by 497
Analyzing incomplete longitudinal clinical trial data
Molenberghs, G. (2004) · Biostatistics · doi:10.1093/biostatistics/5.3.445 · Cited by 463
Hayati Rezvan, P., Lee, K. J., & Simpson, J. A. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0022-1 · Cited by 459
Self-citationGraham, J. W. (2012) · CERN Document Server (European Organization for Nuclear Research) · doi:10.1007/978-1-4614-4018-5 · Cited by 451
Multiple imputation: review of theory, implementation and software
Harel, O., & Zhou, X. (2007) · Statistics in Medicine · doi:10.1002/sim.2787 · Cited by 427
Dealing With Missing Outcome Data in Randomized Trials and Observational Studies
Groenwold, R. H. H., Donders, A. R. T., Roes, K. C. B., Harrell, F. E., & Moons, K. G. M. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kwr302 · Cited by 398
Handling missing data in RCTs; a review of the top medical journals
Bell, M. L., Fiero, M., Horton, N. J., & Hsu, C.-H. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-118 · Cited by 395 · cited 1×
“Research articles and guidance documents have emphasized the role of prevention in minimizing the impact of missing data, but most randomized controlled trials (RCTs) have some missing data.(5) Given the potential for biased and inefficient treatment effect estimates, it is crucial that missing data are handled approp…”
BackgroundAdjusting for partially missing baseline measurements in randomized trials
White, I. R., & Thompson, S. G. (2005) · Statistics in Medicine · doi:10.1002/sim.1981 · Cited by 378
Self-citationIncluding all individuals is not enough: Lessons for intention-to-treat analysis
White, I. R., Carpenter, J., & Horton, N. J. (2012) · Clinical Trials · doi:10.1177/1740774512450098 · Cited by 305
Self-citationStrategies for dealing with missing data in clinical trials: from design to analysis.
(2013) · PubMed Central · Cited by 288
Carpenter, J. R., Roger, J. H., & Kenward, M. G. (2013) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2013.834911 · Cited by 286 · cited 11×
“The ability to incorporate auxiliary variables, both for univariate and multivariate outcomes, is considered one of the key strengths of MI.45 Less well known is that LMMs can also benefit from auxiliary variables through joint modelling with the outcome.9,46 Using model (2) for illustration, Z could be an auxiliary v…”
MethodInfluentialPractical and statistical issues in missing data for longitudinal patient-reported outcomes
Bell, M. L., & Fairclough, D. L. (2014) · Statistical Methods in Medical Research · doi:10.1177/0962280213476378 · Cited by 285
Missing data in randomised controlled trials: a practical guide
Carpenter, J. R., & Kenward, M. (2007) · LSHTM Research Online (London School of Hygiene and Tropical Medicine) · Cited by 285
Inclusion of patients in clinical trial analysis: the intention‐to‐treat principle
Heritier, S. R., Gebski, V. J., & Keech, A. C. (2003) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2003.tb05627.x · Cited by 270
Hernández, A. V., Steyerberg, E. W., & Habbema, J. D. F. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.09.014 · Cited by 258
The use and reporting of multiple imputation in medical research – a review
Mackinnon, A. (2010) · Journal of Internal Medicine · doi:10.1111/j.1365-2796.2010.02274.x · Cited by 229
Graham, J. W., & Donaldson, S. I. (1993) · Journal of Applied Psychology · doi:10.1037//0021-9010.78.1.119 · Cited by 202
Missing outcomes in randomized trials: addressing the dilemma.
Altman, D. G. (2009) · PubMed Central · Cited by 192
Knol, M. J., Janssen, K. J. M., Donders, A. R. T., Egberts, A. C. G., Heerdink, E. R., Grobbee, D. E., Moons, K. G. M., & Geerlings, M. I. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.08.028 · Cited by 189
Mazza, G. L., Enders, C. K., & Ruehlman, L. S. (2015) · Multivariate Behavioral Research · doi:10.1080/00273171.2015.1068157 · Cited by 180
Ratitch, B., O'Kelly, M., & Tosiello, R. (2013) · Pharmaceutical Statistics · doi:10.1002/pst.1549 · Cited by 155
Beunckens, C., Molenberghs, G., & Kenward, M. G. (2005) · Clinical Trials · doi:10.1191/1740774505cn119oa · Cited by 154
Mallinckrodt, C. H., Clark, W. S., Carroll, R. J., & Molenberghs, G. (2003) · Journal of Biopharmaceutical Statistics · doi:10.1081/bip-120019265 · Cited by 138
Alshurafa, M., Briel, M., Akl, E. A., Haines, T., Moayyedi, P., Gentles, S. J., Rios, L., Tran, C., Bhatnagar, N., Lamontagne, F., et al. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049163 · Cited by 126 · cited 1×
“Exactly what constitutes an ITT analysis in the presence of missing data has been much debated.(28) Some researchers have suggested that missing outcome data ought to be imputed, so that the full randomized sample can be included in the analysis.”
BackgroundAddressing Missing Data in Clinical Trials
Fleming, T. R. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-154-2-201101180-00010 · Cited by 120
Recovery of information from multiple imputation: a simulation study
Lee, K. J., & Carlin, J. B. (2012) · Emerging Themes in Epidemiology · doi:10.1186/1742-7622-9-3 · Cited by 113
Self-citationCollins, C. T., Gibson, R. A., Anderson, P. J., McPhee, A. J., Sullivan, T., Gould, J. F., Ryan, P., Doyle, L. W., Davis, P. G., McMichael, J., et al. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-007314 · Cited by 108
Self-citationSullivan, T. R., Salter, A. B., Ryan, P., & Lee, K. J. (2015) · American Journal of Epidemiology · doi:10.1093/aje/kwv100 · Cited by 86
Self-citationA consequence of omitted covariates when estimating odds ratios
Hauck, W. W., Neuhaus, J. M., Kalbfleisch, J. D., & Anderson, S. (1991) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(91)90203-l · Cited by 83
The design and conduct of clinical trials to limit missing data
Little, R. J. A., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Neaton, J. D., Shih, W. J., Siegel, J. P., & Stern, H. S. (2012) · Statistics in Medicine · doi:10.1002/sim.5519 · Cited by 69
Specification of covariance structure in longitudinal data analysis for randomized clinical trials
Lu, K., & Mehrotra, D. V. (2009) · Statistics in Medicine · doi:10.1002/sim.3820 · Cited by 56
Intention‐to‐treat analysis with treatment discontinuation and missing data in clinical trials
Little, R. J. A., & Kang, S. (2014) · Statistics in Medicine · doi:10.1002/sim.6352 · Cited by 53
Collins, C. T., Makrides, M., Gibson, R. A., McPhee, A. J., Davis, P. G., Doyle, L. W., Simmer, K. N., Colditz, P. B., Morris, S. A., Sullivan, T., et al. (2011) · British Journal Of Nutrition · doi:10.1017/s000711451000509x · Cited by 48
Self-citationThe Effects of Auxiliary Variables on Coefficient Bias and Efficiency in Multiple Imputation
Mustillo, S. (2012) · Sociological Methods & Research · doi:10.1177/0049124112452392 · Cited by 26
Correction of bias from non‐random missing longitudinal data using auxiliary information
Wang, C., & Hall, C. B. (2009) · Statistics in Medicine · doi:10.1002/sim.3821 · Cited by 25
Engels, E. (2000) · doi:10.1002/1097-0258(20000715)19:13 · Cited by 2
Research works printed but not linked: 9
- National Research Council, Panel on Handling Missing Data in Clinical Trials, Committee on National Statistics, Division of Behavioral and Social Sciences and Education. The prevention and treatment of missing data in clinical trials . Washington, DC: National Academies Press, 2010.
- 10.1093/biostatistics/kxh001 · DOI · Paper page
- Raghunathan T, 2001, Survey Methodol, 27, 85
- 10.1002/sim.6841 · doi:10.1002/sim.6841
- Altman DG, 2009, Open Med, 3, e51
- Dziura JD, 2013, Yale J Biol Med, 86, 343
- 10.1037/0021-9010.78.1.119 · doi:10.1037/0021-9010.78.1.119 · Paper page
- 10.4310/SII.2013.v6.n3.a2 · doi:10.4310/sii.2013.v6.n3.a2
- 10.1002/1097-0258(20000715)19:13<1793::AID-SIM482>3.0.CO;2-Q · doi:10.1002/1097-0258(20000715)19:13<1793::aid-sim482>3.0.co;2-q
Cited by
Missing Data in Clinical Research: A Tutorial on Multiple Imputation
Austin, P. C., White, I. R., Lee, D. S., & van Buuren, S. (2021) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2020.11.010 · Cited by 1,006
Self-citationBaker, C. W., Glassman, A. R., Beaulieu, W. T., Antoszyk, A. N., Browning, D. J., Chalam, K. V., Grover, S., Jampol, L. M., Jhaveri, C. D., Melia, M., et al. (2019) · JAMA · doi:10.1001/jama.2019.5790 · Cited by 299
Verhoeven, J. E., Han, L. K. M., Lever-van Milligen, B. A., Hu, M. X., Révész, D., Hoogendoorn, A. W., Batelaan, N. M., van Schaik, D. J. F., van Balkom, A. J. L. M., van Oppen, P., et al. (2023) · Journal of Affective Disorders · doi:10.1016/j.jad.2023.02.064 · Cited by 112
Planning a method for covariate adjustment in individually randomised trials: a practical guide
Morris, T. P., Walker, A. S., Williamson, E. J., & White, I. R. (2022) · Trials · doi:10.1186/s13063-022-06097-z · Cited by 97
“With outcomes missing according to this mechanism (depending on covariates), performing multiple imputation by-randomised-arm and then analysing the imputed data sets by any of the three approaches to adjustment targets the all-randomised population [39].”
MethodSelf-citationHajek, P., Przulj, D., Pesola, F., Griffiths, C., Walton, R., McRobbie, H., Coleman, T., Lewis, S., Whitemore, R., Clark, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01808-0 · Cited by 85
Metadata
- Authors
- Sullivan, T. R., White, I. R., Salter, A. B., Ryan, P., & Lee, K. J.
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2016-12-19
- Volume
- 27
- Issue
- 9
- Pages
- 2610-2626
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2574527709
- Cited by
- 308
- References
- 59
- FWCI
- 8.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.