Missing Data in Alcohol Clinical Trials: A Comparison of Methods
Open accessAbstract
BACKGROUND: The rate of participant attrition in alcohol clinical trials is often substantial and can cause significant issues with regard to the handling of missing data in statistical analyses of treatment effects. It is common for researchers to assume that missing data is indicative of participant relapse, and under that assumption, many researchers have relied on setting all missing values to the worst-case scenario for the outcome (e.g., missing = heavy drinking). This sort of single-imputation method has been criticized for producing biased results in other areas of clinical research, but has not been evaluated within the context of alcohol clinical trials, and many alcohol researchers continue to use the missing = heavy drinking assumption. METHODS: Data from the COMBINE study, a multisite randomized clinical trial, were used to generate simulated situations of missing data under a variety of conditions and assumptions. We manipulated the sample size (n = 200, 500, and 1,000) and dropout rate (5, 10, 25, 30%) under 3 missing data assumptions (missing completely at random, missing at random, and missing not at random). We then examined the association between receiving naltrexone and heavy drinking during the first 10 weeks following treatment using 5 methods for treating missing data (complete case analysis [CCA], last observation carried forward [LOCF], missing = heavy drinking, multiple imputation [MI], and full information maximum likelihood [FIML]). RESULTS: CCA, LOCF, and missing = heavy drinking produced the most biased naltrexone effect estimates and standard errors under conditions that are likely to exist in randomized clinical trials. MI and FIML produced the least biased naltrexone effect estimates and standard errors. CONCLUSIONS: Assuming that missing = heavy drinking produces biased results of the treatment effect and should not be used to evaluate treatment effects in alcohol clinical trials.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Chiesa et al. presents somewhat low risk of bias, but missing data imputed using last observation carried forward may introduce bias.
“Overall this study presents somewhat low risk of bias (see Table 2), but missing data was imputed using the last observation carried forward method, which may introduce bias (Hallgren & Witkiewitz, 2013).”
From Dismantling Mindfulness-Based Programs: a Systematic Review to Identify Active Components of Treatment (2020) · cites this paper as Hallgren & Witkiewitz, 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
<b>mice</b>: Multivariate Imputation by Chained Equations in<i>R</i>
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · see citing works
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638 · see citing works
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925 · see citing works
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473 · see citing works
Missing data analysis: making it work in the real world
Graham, J. W. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085530 · Cited by 6,163 · see citing works
(1997) · American Psychiatric Press eBooks · Cited by 3,056 · see citing works
A comparison of inclusive and restrictive strategies in modern missing data procedures.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.330 · Cited by 2,389 · see citing works
Anton, R. F., O’Malley, S. S., Ciraulo, D. A., Cisler, R. A., Couper, D., Donovan, D. M., Gastfriend, D. R., Hosking, J. D., Johnson, B. A., LoCastro, J. S., et al. (2006) · JAMA · doi:10.1001/jama.295.17.2003 · Cited by 1,930 · see citing works
(1997) · Journal of Studies on Alcohol · doi:10.15288/jsa.1997.58.7 · Cited by 1,905 · see citing works
The prevention and treatment of missing data in clinical trials
Little, R. J., D'Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Frangakis, C., Hogan, J. W., Molenberghs, G., Murphy, S. A., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmsr1203730 · Cited by 1,884 · see citing works
An introduction to modern missing data analyses
Baraldi, A. N., & Enders, C. K. (2009) · Journal of School Psychology · doi:10.1016/j.jsp.2009.10.001 · Cited by 1,464 · see citing works
Application of random-effects pattern-mixture models for missing data in longitudinal studies.
Hedeker, D., & Gibbons, R. D. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.1.64 · Cited by 907 · see citing works
The design of simulation studies in medical statistics
Burton, A., Altman, D. G., Royston, P. J., & HOLDER, R. L. (2006) · Statistics in Medicine · doi:10.1002/sim.2673 · Cited by 755 · see citing works
Johnson, B. A. (2007) · JAMA · doi:10.1001/jama.298.14.1641 · Cited by 542 · see citing works
Wu, M. C., & Carroll, R. J. (1988) · Biometrics · doi:10.2307/2531905 · Cited by 539 · see citing works
Wu, M. C., & Carroll, R. J. (1987) · doi:10.21236/ada186320 · Cited by 526 · see citing works
Analyzing incomplete longitudinal clinical trial data
Molenberghs, G. (2004) · Biostatistics · doi:10.1093/biostatistics/5.3.445 · Cited by 463 · see citing works
Measurement of drinking behavior using the Form 90 family of instruments.
Miller, W. R., & Boca, F. K. D. (1994) · Journal of Studies on Alcohol Supplement · doi:10.15288/jsas.1994.s12.112 · Cited by 450 · see citing works
The Prevention and Treatment of Missing Data in Clinical Trials
National Research Council (2010) · National Academies Press eBooks · doi:10.17226/12955 · Cited by 439 · see citing works
ACCOUNTING FOR DROPOUT BIAS USING MIXED-EFFECTS MODELS
Mallinckrodt, C. H., Clark, W. S., & David, S. R. (2001) · Journal of Biopharmaceutical Statistics · doi:10.1081/bip-100104194 · Cited by 398 · cited 18× · see citing works
“The current findings are consistent with prior simulation studies of missing data in clinical trials (e.g., Lane, 2008; Mallinckrodt et al., 2001) and we echo the recommendations of many others to use MI, FIML, or other recommended modern missing data analytic tools (e.g., weighted estimating…”
BackgroundMethodResultInfluentialAnalyzing incomplete longitudinal clinical trial data
Molenberghs, G. (2004) · Biostatistics · doi:10.1093/biostatistics/kxh001 · Cited by 378 · see citing works
Wu, M. C., & Bailey, K. R. (1989) · Biometrics · doi:10.2307/2531694 · Cited by 372 · see citing works
Recommendations for the Primary Analysis of Continuous Endpoints in Longitudinal Clinical Trials
Mallinckrodt, C. H., Lane, P. W., Schnell, D. J., Peng, Y., & Mancuso, J. P. (2008) · Drug Information Journal · doi:10.1177/009286150804200402 · Cited by 358 · see citing works
MMRM vs. LOCF: A Comprehensive Comparison Based on Simulation Study and 25 NDA Datasets
Siddiqui, O., Hung, H. M. J., & O'Neill, R. (2009) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543400802609797 · Cited by 301 · see citing works
Handling drop‐out in longitudinal clinical trials: a comparison of the LOCF and MMRM approaches
Lane, P. (2008) · Pharmaceutical Statistics · doi:10.1002/pst.267 · Cited by 262 · cited 18× · see citing works
“The current findings are consistent with prior simulation studies of missing data in clinical trials (e.g., Lane, 2008; Mallinckrodt et al., 2001) and we echo the recommendations of many others to use MI, FIML, or other recommended modern missing data analytic tools (e.g., weighted estimating…”
BackgroundMethodResultInfluentialMissing not at random models for latent growth curve analyses.
Enders, C. K. (2011) · Psychological Methods · doi:10.1037/a0022640 · Cited by 255 · see citing works
Falk, D. E., Wang, X. Q., Liu, L., Fertig, J. B., Mattson, M. E., Ryan, M., Johnson, B. A., Stout, R. L., & Litten, R. Z. (2010) · Alcoholism Clinical and Experimental Research · doi:10.1111/j.1530-0277.2010.01290.x · Cited by 189 · see citing works
The COMBINE Study Research Group (2003) · Alcoholism: Clinical and Experimental Research · doi:10.1111/j.1530-0277.2003.tb02873.x · Cited by 174 · see citing works
Hedeker, D., Mermelstein, R. J., & Demirtaş, H. (2007) · Addiction · doi:10.1111/j.1360-0443.2007.01946.x · Cited by 154 · see citing works
COMPARISON OF ALTERNATIVE STRATEGIES FOR ANALYSIS OF LONGITUDINAL TRIALS WITH DROPOUTS
Liu, G. F., & Gould, A. L. (2002) · Journal of Biopharmaceutical Statistics · doi:10.1081/bip-120015744 · Cited by 101 · see citing works
Attrition in web-based treatment for problem drinkers
Postel, M. G., de Haan, H. A., ter Huurne, E. D., van der Palen, J., Becker, E. S., & de Jong, C. A. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1811 · Cited by 87 · see citing works
MMRM versus MI in Dealing with Missing Data—A Comparison Based on 25 NDA Data Sets
Siddiqui, O. (2011) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543401003777995 · Cited by 63 · see citing works
Fertig, J. B., Ryan, M. L., Falk, D. E., Litten, R. Z., Mattson, M. E., Ransom, J. H., Rickman, W. J., Scott, C., Ciraulo, D. A., Green, A. I., et al. (2012) · Alcoholism Clinical and Experimental Research · doi:10.1111/j.1530-0277.2011.01716.x · Cited by 53 · see citing works
Missing data assumptions and methods in a smoking cessation study
Barnes, S. A., Larsen, M. D., Schroeder, D. R., Hanson, A. C., & Decker, P. A. (2010) · Addiction · doi:10.1111/j.1360-0443.2009.02809.x · Cited by 51 · see citing works
The characteristics of alcoholics frequently lost to follow-up.
Mackenzie, A., Funderburk, F. R., Allen, R. P., et al. (1987) · Journal of Studies on Alcohol · doi:10.15288/jsa.1987.48.119 · Cited by 46 · see citing works
Analytical approaches to reporting long-term clinical trial data
Papp, K. A., Fonjallaz, P., Casset‐Semanaz, F., Krueger, J. G., & Wittkowski, K. M. (2008) · Current Medical Research and Opinion · doi:10.1185/03007990802215315 · Cited by 45 · see citing works
Conducting Simulation Studies in the R Programming Environment
Hallgren, K. A. (2013) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.09.2.p043 · Cited by 40 · see citing works
Self-citationSuh, J. J., Pettinati, H., Kampman, K., & O'Brien, C. (2008) · American Journal on Addictions · doi:10.1080/10550490802409074 · Cited by 33 · see citing works
Hedden, S. L., Woolson, R. F., Carter, R. E., Palesch, Y. Y., Upadhyaya, H. P., & Malcolm, R. J. (2008) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2008.09.011 · Cited by 31 · see citing works
Follow-up attrition in alcohol treatment studies: Is ‘no news’ bad news, good news or no news?
Sobell, L. C., Sobell, M. B., & Maisto, S. A. (1984) · Drug and Alcohol Dependence · doi:10.1016/0376-8716(84)90027-9 · Cited by 19 · see citing works
Stereotyping and the treatment of missing data for drug and alcohol clinical trials
Arndt, S. V. (2009) · Substance Abuse Treatment Prevention and Policy · doi:10.1186/1747-597x-4-2 · Cited by 19 · see citing works
Prisciandaro, J. J., Rembold, J., Brown, D. G., Brady, K. T., & Tolliver, B. K. (2011) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2011.03.029 · Cited by 18 · see citing works
Witkiewitz, K. A., Bush, T. M., Magnusson, B., et al. (2011) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntr291 · Cited by 8 · see citing works
Self-citationZhang, W., & Yung, Y. (2011) · Cited by 2 · see citing works
Multiple imputation of missing blood pressure covariates in survival analysis
van Buuren, S., Boshuizen, H. C., & Knook, D. L. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990330)18:6
Flagged first, then by citations
Research works printed but not linked: 1
- FDA 2006 Medical Review of Vivitrol: 21-897 US Government Rockville, Maryland
Cited by
Sorted by citations
The Use and Effectiveness of Mobile Apps for Depression: Results From a Fully Remote Clinical Trial
Arean, P. A., Hallgren, K. A., Jordan, J. T., Gazzaley, A., Atkins, D. C., Heagerty, P. J., & Anguera, J. A. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.6482 · Cited by 376 · see citing works
Self-citationBricker, J. B., Mull, K. E., Kientz, J. A., Vilardaga, R., Mercer, L. D., Akioka, K. J., & Heffner, J. L. (2014) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2014.07.006 · Cited by 340 · see citing works
Klausen, M. K., Jensen, M. E., Møller, M., Le Dous, N., Jensen, A.-M. Ø., Zeeman, V. A., Johannsen, C.-F., Lee, A., Thomsen, G. K., Macoveanu, J., et al. (2022) · JCI Insight · doi:10.1172/jci.insight.159863 · Cited by 262 · see citing works
Stein, E., & Witkiewitz, K. (2020) · Mindfulness · doi:10.1007/s12671-020-01444-0 · Cited by 63 · see citing works
“Overall, this study presents somewhat low risk of bias (see Table 2), but missing data was imputed using the last observation carried forward method, which may introduce bias (Hallgren and Witkiewitz 2013).”
MethodSelf-citationWitkiewitz, K., Falk, D. E., Kranzler, H. R., Litten, R. Z., Hallgren, K. A., O'Malley, S. S., Anton, R. F., & Workgroup, I. C. W. T. A. C. T. I. (. (2014) · Alcoholism: Clinical and Experimental Research · doi:10.1111/acer.12543 · Cited by 60 · see citing works
Self-citation
- Authors
- Hallgren, K. A., & Witkiewitz, K.
- Affiliations
- University of New Mexico (US)
- Corresponding author
- Hallgren, Kevin A.
- Venue
- Alcoholism: Clinical and Experimental Research
- Publisher
- Wiley
- Published
- 2013-07-24
- Volume
- 37
- Issue
- 12
- Pages
- 2152-2160
- ISSN
- 0145-6008 (ISSN-L)
- PMC
- PMC4113114
- OpenAlex
- W2110117912
- Cited by
- 176
- References
- 45
- Citations by year
- 2026: 10 · 2025: 17 · 2024: 11 · 2023: 12 · 2022: 13 · 2021: 16 · 2020: 23 · 2019: 10 · 2018: 12 · 2017: 12 · 2016: 14 · 2015: 16 · 2014: 8 · 2013: 2
- FWCI
- 11.77
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Alcoholism Clinical and Experimental Research (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute on Alcohol Abuse and Alcoholism (R01AA022328)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- missing data, participant attrition, naltrexone, heavy drinking, multiple imputation, full information maximum likelihood, last observation carried forward, complete case analysis, missing completely at random, missing at random, missing not at random, treatment effect bias, COMBINE study, relapse
- MeSH terms
- Data Collection, Humans, Naltrexone, Narcotic Antagonists, Patient Dropouts, Recurrence, Bias, Randomized Controlled Trials as Topic, Alcohol-Related Disorders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
