Dealing With Missing Outcome Data in Randomized Trials and Observational Studies
Open accessAbstract
Although missing outcome data are an important problem in randomized trials and observational studies, methods to address this issue can be difficult to apply. Using simulated data, the authors compared 3 methods to handle missing outcome data: 1) complete case analysis; 2) single imputation; and 3) multiple imputation (all 3 with and without covariate adjustment). Simulated scenarios focused on continuous or dichotomous missing outcome data from randomized trials or observational studies. When outcomes were missing at random, single and multiple imputations yielded unbiased estimates after covariate adjustment. Estimates obtained by complete case analysis with covariate adjustment were unbiased as well, with coverage close to 95%. When outcome data were missing not at random, all methods gave biased estimates, but handling missing outcome data by means of 1 of the 3 methods reduced bias compared with a complete case analysis without covariate adjustment. Complete case analysis with covariate adjustment and multiple imputation yield similar estimates in the event of missing outcome data, as long as the same predictors of missingness are included. Hence, complete case analysis with covariate adjustment can and should be used as the analysis of choice more often. Multiple imputation, in addition, can accommodate the missing-not-at-random scenario more flexibly, making it especially suited for sensitivity analyses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Missing data will be imputed with multiple imputation techniques.
“Missing data will be imputed with multiple imputation techniques [58, 59].”
Some argue that using covariates yields similar results to imputing, but this may depend heavily on the dataset, and multiple imputation remains the solution of choice for missing data.
“Although some argue that using covariates yields similar results to imputing [67], this may depend heavily on the dataset, and multiple imputation remains the solution of choice for missing data [68].”
From Short-term effectiveness of web-based guided self-help for phobic outpatients: randomized controlled trial. (2014) · cites this paper as [67]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701 · see citing works
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,384 · see citing works
<b>mice</b>: Multivariate Imputation by Chained Equations in<i>R</i>
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · see citing works
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638 · see citing works
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473 · see citing works
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299 · see citing works
Analysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012 · see citing works
Analysis of Incomplete Multivariate Data
Booth, D. E., & Schafer, J. L. (2000) · Technometrics · doi:10.2307/1271466 · Cited by 5,627 · see citing works
Review: a gentle introduction to imputation of missing values.
Donders, A. R. T., van der Heijden, G. J. M. G., Stijnen, T., & Moons, K. G. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.014 · Cited by 2,582 · see citing works
Self-citationA comparison of inclusive and restrictive strategies in modern missing data procedures.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.330 · Cited by 2,389 · see citing works
Pocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002) · Statistics in Medicine · doi:10.1002/sim.1296 · Cited by 1,077 · see citing works
Using the outcome for imputation of missing predictor values was preferred
Moons, K. G. M., Donders, R. A. R. T., Stijnen, T., & Harrell, F. E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.009 · Cited by 994 · see citing works
Self-citationRegression with Missing X 's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10476282 · Cited by 968 · see citing works
A critical look at methods for handling missing covariates in epidemiologic regression analyses.
Greenland, S., & Finkle, W. D. (1995) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117592 · Cited by 891 · see citing works
Strategy for intention to treat analysis in randomised trials with missing outcome data
White, I. R., Horton, N. J., Carpenter, J. R., statistics, R. I. M. A. S., & Pocock, S. (2011) · BMJ · doi:10.1136/bmj.d40 · Cited by 849 · see citing works
Gail, M. H., Wieand, S., & Piantadosi, S. (1984) · Biometrika · doi:10.1093/biomet/71.3.431 · Cited by 683 · see citing works
Regression With Missing X's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.2307/2290664 · Cited by 647 · see citing works
Analyzing incomplete longitudinal clinical trial data
Molenberghs, G. (2004) · Biostatistics · doi:10.1093/biostatistics/5.3.445 · Cited by 463 · see citing works
Statistical Issues in Drug Development
Senn, S. (2007) · doi:10.1002/9780470723586 · Cited by 448 · see citing works
Statistical Issues in Drug Development
Senn, S. J. (2021) · Wiley eBooks · doi:10.1002/9781119238614 · Cited by 384 · see citing works
Adjusting for partially missing baseline measurements in randomized trials
White, I. R., & Thompson, S. G. (2005) · Statistics in Medicine · doi:10.1002/sim.1981 · Cited by 378 · see citing works
Analyzing incomplete longitudinal clinical trial data
Molenberghs, G. (2004) · Biostatistics · doi:10.1093/biostatistics/kxh001 · Cited by 378 · see citing works
Recommendations for the Primary Analysis of Continuous Endpoints in Longitudinal Clinical Trials
Mallinckrodt, C. H., Lane, P. W., Schnell, D. J., Peng, Y., & Mancuso, J. P. (2008) · Drug Information Journal · doi:10.1177/009286150804200402 · Cited by 358 · see citing works
Imputation methods for missing outcome data in meta-analysis of clinical trials
Higgins, J. P. T., White, I. R., & Wood, A. (2008) · Clinical Trials · doi:10.1177/1740774508091600 · Cited by 352 · see citing works
Tsiatis, A. A., Davidian, M., Zhang, M., & Lu, X. (2007) · Statistics in Medicine · doi:10.1002/sim.3113 · Cited by 350 · see citing works
Missing data in randomised controlled trials: a practical guide
Carpenter, J. R., & Kenward, M. (2007) · LSHTM Research Online (London School of Hygiene and Tropical Medicine) · Cited by 204 · see citing works
Austin, P. C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.11.004 · Cited by 197 · see citing works
Altman, D. G., & Bland, M. (2007) · BMJ · doi:10.1136/bmj.38977.682025.2c · Cited by 196 · see citing works
Missing outcomes in randomized trials: addressing the dilemma.
Altman, D. G. (2009) · PubMed Central · Cited by 149 · see citing works
(2004) · Statistics in Medicine · doi:10.1002/sim.1647 · Cited by 122 · see citing works
Logistic Regression with Missing Values in the Covariates
Vach, W. (1994) · Lecture notes in statistics · doi:10.1007/978-1-4612-2650-5 · Cited by 82 · see citing works
Reporting of treatment effects from randomized trials: A plea for multivariable risk ratios
Groenwold, R. H. H., Moons, K. G. M., Peelen, L. M., Knol, M. J., & Hoes, A. W. (2011) · Contemporary Clinical Trials · doi:10.1016/j.cct.2010.12.011 · Cited by 27 · see citing works
Self-citationTan, M. T., Qu, Y., Mascha, E. J., & Schubert, A. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990815)18:15 · Cited by 26 · see citing works
Baker, S. G. (2005) · Biostatistics · doi:10.1093/biostatistics/kxi038 · Cited by 19 · see citing works
Editorial: Dealing with the Missing Data Challenge in Clinical Trials
Permutt, T., & Pinheiro, J. C. (2009) · Drug Information Journal · doi:10.1177/009286150904300403 · Cited by 8 · see citing works
Flyer, P. A. (2009) · Drug Information Journal · doi:10.1177/009286150904300407 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Lewis JA. Statistical principles for clinical trials (ICH E9): an introductory note on an international guideline. Stat Med. 1999; 18(15):1903-1942.
Cited by
Sorted by citations
SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251 · see citing works
A Trial of Sugar-free or Sugar-Sweetened Beverages and Body Weight in Children
de Ruyter, J. C., Olthof, M. R., Seidell, J. C., & Katan, M. B. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1203034 · Cited by 814 · see citing works
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · see citing works
Self-citationKuramatsu, J. B., Gerner, S. T., Schellinger, P. D., Glahn, J., Endres, M., Sobesky, J., Flechsenhar, J., Neugebauer, H., Jüttler, E., Grau, A., et al. (2015) · JAMA · doi:10.1001/jama.2015.0846 · Cited by 568 · see citing works
Mediterranean dietary pattern and depression: the PREDIMED randomized trial
Sánchez-Villegas, A., Martínez-González, M. A., Estruch, R., Salas-Salvadó, J., Corella, D., Covas, M. I., Arós, F., Romaguera, D., Gómez-Gracia, E., Lapetra, J., et al. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-208 · Cited by 403 · see citing works
Handling missing data in clinical research
Heymans, M. W., & Twisk, J. W. R. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.08.016 · Cited by 399 · see citing works
PCSK9 genetic variants and risk of type 2 diabetes: a mendelian randomisation study
Schmidt, A. F., Swerdlow, D. I., Holmes, M. V., Patel, R. S., Fairhurst-Hunter, Z., Lyall, D. M., Hartwig, F. P., Horta, B. L., Hyppönen, E., Power, C., et al. (2017) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(16)30396-5 · Cited by 380 · see citing works
Kennedy, A., Bower, P., Reeves, D., Blakeman, T., Bowen, R., Chew-Graham, C., Eden, M., Fullwood, C., Gaffney, H., Gardner, C., et al. (2013) · BMJ · doi:10.1136/bmj.f2882 · Cited by 239 · see citing works
Outcome-sensitive multiple imputation: a simulation study
Kontopantelis, E., White, I. R., Sperrin, M., & Buchan, I. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0281-5 · Cited by 205 · see citing works
Brazil, K., Carter, G., Cardwell, C., Clarke, M., Hudson, P., Froggatt, K., McLaughlin, D., Passmore, P., & Kernohan, W. G. (2018) · Palliative Medicine · doi:10.1177/0269216317722413 · Cited by 164 · see citing works
Jensen, J. F., Egerod, I., Bestle, M. H., Christensen, D. F., Elklit, A., Hansen, R. L., Knudsen, H., Grode, L. B., & Overgaard, D. (2016) · Intensive Care Medicine · doi:10.1007/s00134-016-4522-1 · Cited by 150 · see citing works
Estruch, R., Martínez-González, M. A., Corella, D., Salas-Salvadó, J., Fitó, M., Chiva-Blanch, G., Fiol, M., Gómez-Gracia, E., Arós, F., Lapetra, J., et al. (2019) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(19)30074-9 · Cited by 147 · see citing works
RetractedEbrahim, S., Akl, E. A., Mustafa, R. A., Sun, X., Walter, S. D., Heels-Ansdell, D., Alonso-Coello, P., Johnston, B. C., & Guyatt, G. H. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.014 · Cited by 141 · see citing works
Reduction of bodily pain in response to an online positive activities intervention
Hausmann, L. R. M., Parks, A., Youk, A. O., & Kwoh, C. K. (2014) · The Journal of Pain · doi:10.1016/j.jpain.2014.02.004 · Cited by 82 · see citing works
Goodall, E. C., Granados, A. C., Luinstra, K., Pullenayegum, E., Coleman, B. L., Loeb, M., & Smieja, M. (2014) · BMC Infectious Diseases · doi:10.1186/1471-2334-14-273 · Cited by 75 · see citing works
Ebrahim, S., Johnston, B. C., Akl, E. A., Mustafa, R. A., Sun, X., Walter, S. D., Heels-Ansdell, D., Alonso-Coello, P., & Guyatt, G. H. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.11.014 · Cited by 72 · see citing works
Kok, R. N., van Straten, A., Beekman, A. T. F., & Cuijpers, P. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3429 · Cited by 50 · see citing works
“Although some argue that using covariates yields similar results to imputing [67], this may depend heavily on the dataset, and multiple imputation remains the solution of choice for missing data [68].”
Long, K. N. G., Chen, Y., Potts, M., Hanson, J., & VanderWeele, T. J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01337 · Cited by 33 · see citing works
Approaches to addressing missing values, measurement error, and confounding in epidemiologic studies
van Smeden, M., Penning de Vries, B. B. L., Nab, L., & Groenwold, R. H. H. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.11.006 · Cited by 32 · see citing works
Self-citationVolpato, E., Banfi, P., & Pagnini, F. (2017) · Trials · doi:10.1186/s13063-017-1802-1 · Cited by 18 · see citing works
Pimenta, A. M., Bes-Rastrollo, M., Gea, A., Sayón-Orea, C., Zazpe, I., Lopez-Iracheta, R., & Martinez-Gonzalez, M. A. (2016) · Public Health Nutrition · doi:10.1017/s1368980015001342 · Cited by 18 · see citing works
Re: "Dealing With Missing Outcome Data in Randomized Trials and Observational Studies"
Liublinska, V., & Rubin, D. B. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws215 · Cited by 10 · see citing works
- Authors
- Groenwold, R. H. H., Donders, A. R. T., Roes, K. C. B., Harrell, F. E., & Moons, K. G. M.
- Affiliations
- University Medical Center Utrecht (NL)
- Corresponding author
- Groenwold, Rolf H. H.
- Venue
- American Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2011-12-23
- Volume
- 175
- Issue
- 3
- Pages
- 210-217
- ISSN
- 0002-9262 (ISSN-L)
- 1476-6256
- OpenAlex
- W2097395684
- Cited by
- 398
- References
- 37
- Citations by year
- 2026: 16 · 2025: 26 · 2024: 30 · 2023: 22 · 2022: 37 · 2021: 36 · 2020: 38 · 2019: 25 · 2018: 27 · 2017: 28 · 2016: 35 · 2015: 15 · 2014: 27 · 2013: 23 · 2012: 13
- FWCI
- 17.61
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Funders
- Vanderbilt University
- Radboud Universiteit
- School of Medicine, Vanderbilt University
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- missing outcome data, multiple imputation, single imputation, complete case analysis, missing at random, missing not at random, covariate adjustment, sensitivity analysis, observational studies, estimation bias
- MeSH terms
- Computer Simulation, Data Interpretation, Statistical, Humans, Models, Statistical, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
