The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals
Abstract
Background Randomized controlled trials almost always have some individuals with missing outcomes. Inadequate handling of these missing data in the analysis can cause substantial bias in the treatment effect estimates. We examine how missing outcome data are handled in randomized controlled trials in order to assess whether adequate steps have been taken to reduce nonresponse bias and to identify ways to improve procedures for missing data. Methods We reviewed all randomized trials published between July and December 2001 in BMJ, JAMA, Lancet and New England Journal of Medicine, excluding trials in which the primary outcome was described as a time-to-event. We focused on trial designs, how missing outcome data were described and the statistical methods used to deal with the missing outcome data, including sensitivity analyses. Results We identified 71 trials of which 63 (89%) reported having partly missing outcome data: 13 trials had more than 20% of patients with missing outcomes. In 26 trials that measured the outcome at a single time point, 92% performed a complete case analysis and 8% imputed the missing outcomes using baseline values or the worst case value. In 37 trials with repeated measures of the outcome, 46% performed complete case analyses, potentially excluding individuals with some follow-up data, while 14% performed a repeated measures analysis, 19% used the last observation carried forward, 11% imputed with the worst case value and 2% imputed using regression predictions. Thirteen (21%) of trials with missing data reported a sensitivity analysis. Conclusions Our review shows that missing outcome data are a common problem in randomized controlled trials, and are often inadequately handled in the statistical analysis in the top tier medical journals. Authors should explicitly state the assumptions underlying the handling of the missing outcomes and justify them through data descriptions and sensitivity analyses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
Practical Statistics for Medical Research.
Walter, S. D., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,668
Analysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012 · cited 1×
“Multiple imputation uses several stochastic imputations which gives valid standard errors (under MAR) that incorporate the uncertainty about the imputed value [31].”
MethodAnalysis of Incomplete Multivariate Data
Booth, D. E. (2000) · Technometrics · doi:10.1080/00401706.2000.10486013 · Cited by 5,627
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373
Practical Statistics for Medical Research.
Bartko, J. J., & Altman, D. G. (1992) · Journal of the American Statistical Association · doi:10.2307/2290246 · Cited by 4,797
What is meant by intention to treat analysis? Survey of published randomised controlled trials
Hollis, S., & Campbell, F. (1999) · BMJ · doi:10.1136/bmj.319.7211.670 · Cited by 2,160
Statistical Analysis with Missing Data
Bo., N., Little, R. J. A., & Rubin, D. B. (1988) · Population (French Edition) · doi:10.2307/1533221 · Cited by 2,095
Charbonneau, D. H. (2005) · Europe PMC (PubMed Central) · Cited by 2,041
Albert, M. A., Danielson, E., Rifai, N., Ridker, P. M., & for the PRINCE Investigators (2001) · JAMA · doi:10.1001/jama.286.1.64 · Cited by 1,641
Unbalanced Repeated-Measures Models with Structured Covariance Matrices
Jennrich, R. I., & Schluchter, M. D. (1986) · Biometrics · doi:10.2307/2530695 · Cited by 1,325
Joint modelling of longitudinal measurements and event time data
Henderson, R. (2000) · Biostatistics · doi:10.1093/biostatistics/1.4.465 · Cited by 913
A critical look at methods for handling missing covariates in epidemiologic regression analyses.
Greenland, S., & Finkle, W. D. (1995) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117592 · Cited by 891 · cited 1×
“These methods may introduce bias in observational studies [36] but not in randomized controlled trials [2].”
MethodSample size slippages in randomised trials: exclusions and the lost and wayward.
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07882-0 · Cited by 642
Attrition in longitudinal studies
Twisk, J., & de Vente, W. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00476-0 · Cited by 527 · cited 1×
“There is a large literature comparing methods to handle missing outcomes [7-10] and developing complex statistical procedures for handling MNAR situations and performing sensitivity analyses”
BackgroundSurvival plots of time-to-event outcomes in clinical trials: good practice and pitfalls
Pocock, S. J., Clayton, T. C., & Altman, D. G. (2002) · The Lancet · doi:10.1016/s0140-6736(02)08594-x · Cited by 513
Intent-to-Treat Analysis for Longitudinal Studies with Drop-Outs
Little, R., & Yau, L. (1996) · Biometrics · doi:10.2307/2532847 · Cited by 380
Methods for the analysis of informatively censored longitudinal data
Schluchter, M. D. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111408 · Cited by 287
Long-term Weight Loss With Sibutramine: A Randomized Controlled Trial
Wirth, A., & Krause, J. (2001) · JAMA · doi:10.1001/jama.286.11.1331 · Cited by 272
Unnebrink, K., & Windeler, J. (2001) · Statistics in Medicine · doi:10.1002/sim.1149 · Cited by 258
An introduction to hierarchical linear modelling
Sullivan, L. M., Dukes, K. A., & Losina, E. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990415)18:7<855::aid-sim117>3.0.co;2-7 · Cited by 250
MODEL-BASED APPROACHES TO ANALYSING INCOMPLETE LONGITUDINAL AND FAILURE TIME DATA
Hogan, J. W., & Laird, N. M. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19970215)16:3<259::aid-sim484>3.0.co;2-s · Cited by 243
The Case of the Missing Data: Methods of Dealing with Dropouts and other Research Vagaries
Streiner, D. V. L. (2002) · The Canadian Journal of Psychiatry · doi:10.1177/070674370204700111 · Cited by 215
Interpreting Incomplete Data in Studies of Diet and Weight Loss
Ware, J. H. (2003) · New England Journal of Medicine · doi:10.1056/nejme030054 · Cited by 205
Problems in dealing with missing data and informative censoring in clinical trials
Shih, W. J. (2002) · Current Controlled Trials in Cardiovascular Medicine · doi:10.1186/1468-6708-3-4 · Cited by 170
Worst-Rank Score Analysis with Informatively Missing Observations in Clinical Trials
Lachin, J. M. (1999) · Controlled Clinical Trials · doi:10.1016/s0197-2456(99)00022-7 · Cited by 164
Handling Missing Data in Clinical Trials: An Overview
Myers, W. R. (2000) · Drug Information Journal · doi:10.1177/009286150003400221 · Cited by 120
Mazumdar, S., Liu, K. S., Houck, P. R., & Reynolds, C. F. (1999) · Journal of Psychiatric Research · doi:10.1016/s0022-3956(98)00058-2 · Cited by 103
Sensitivity Analysis for Incomplete Contingency Tables: The Slovenian Plebiscite Case
Molenberghs, G., Kenward, M. G., & Goetghebeur, E. J. T. (2001) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00217 · Cited by 97
Coping with missing data in clinical trials: A model‐based approach applied to asthma trials
Carpenter, J. R., Pocock, S., & Lamm, C. J. (2002) · Statistics in Medicine · doi:10.1002/sim.1065 · Cited by 89
Sensitivity analysis for incomplete categorical data
Kenward, M. G., Goetghebeur, E. J. T., & Molenberghs, G. (2001) · Statistical Modelling · doi:10.1177/1471082x0100100104 · Cited by 77
Correcting for the bias caused by drop‐outs in hypertension trials
Murray, G. D., & Findlay, J. G. (1988) · Statistics in Medicine · doi:10.1002/sim.4780070905 · Cited by 73
Touloumi, G., Babiker, A. G., Pocock, S., & Darbyshire, J. H. (2001) · Statistics in Medicine · doi:10.1002/sim.1114 · Cited by 61
Sensitivity Analysis by Worst and Best Case Assessment: Is it Really Sensitive?
Unnebrink, K., & Windeler, J. (1999) · Drug Information Journal · doi:10.1177/009286159903300324 · Cited by 24
Handling Missing Data by Re-approaching Non-respondents
Huisman, M., Krol, B., & van Sonderen, E. (1998) · Quality & Quantity · doi:10.1023/a:1004338522505 · Cited by 18
Cited by
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299
Self-citationSPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251
(2008) · Cited by 3,790
Metadata
- Authors
- Wood, A. M., White, I. R., & Thompson, S. G.
- Venue
- Clinical Trials
- Publisher
- SAGE Publications
- Published
- 2004-08
- Volume
- 1
- Issue
- 4
- Pages
- 368-376
- ISSN
- 1740-7745 (ISSN-L)
- 1740-7753
- OpenAlex
- W2115176304
- Cited by
- 543
- References
- 37
- FWCI
- 5.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.