Combining multiple imputation and bootstrap in the analysis of cost‐effectiveness trial data
Open accessAbstract
In healthcare cost‐effectiveness analysis, probability distributions are typically skewed and missing data are frequent. Bootstrap and multiple imputation are well‐established resampling methods for handling skewed and missing data. However, it is not clear how these techniques should be combined. This paper addresses combining multiple imputation and bootstrap to obtain confidence intervals of the mean difference in outcome for two independent treatment groups. We assessed statistical validity and efficiency of 10 candidate methods and applied these methods to a clinical data set. Single imputation nested in the bootstrap percentile method (with added noise to reflect the uncertainty of the imputation) emerged as the method with the best statistical properties. However, this method can require extensive computation times and the lack of standard software makes this method not accessible for a larger group of researchers. Using a standard unpaired t‐test with standard multiple imputation without bootstrap appears to be a robust alternative with acceptable statistical performance for which standard multiple imputation software is available.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Brand et al. found single imputation embedded inside a bootstrap showed better statistical properties than other methods when a bootstrap is required for missing data.
“When a bootstrap is required for missing data, Brand et al. found single imputation embedded inside a bootstrap showed better statistical properties than other methods 43 .”
From Correlates of protection against symptomatic and asymptomatic SARS-CoV-2 infection (2021) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230 · see citing works
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701 · see citing works
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638 · see citing works
Multiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · see citing works
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473 · see citing works
(1997) · doi:10.1097/00005650-199711000-00002 · Cited by 4,544 · see citing works
Multiple imputation after 18+ years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476908 · Cited by 4,072 · see citing works
The estimation of a preference-based measure of health from the SF-36.
Brazier, J., Roberts, J., & Deverill, M. (2002) · Journal of Health Economics · doi:10.1016/s0167-6296(01)00130-8 · Cited by 3,164 · see citing works
Multiple Imputation After 18+ Years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.2307/2291635 · Cited by 2,686 · see citing works
The Bootstrap and Edgeworth Expansion
Hall, P. V. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-4384-7 · Cited by 2,067 · see citing works
Flexible Imputation of Missing Data
van Buuren, S. (2012) · doi:10.1201/b11826 · Cited by 2,022 · see citing works
Self-citationThe Importance of the Normality Assumption in Large Public Health Data Sets
Lumley, T., Diehr, P., Emerson, S., & Chen, L. (2002) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.23.100901.140546 · Cited by 1,803 · see citing works
US Valuation of the EQ-5D Health States
Shaw, J. W., Johnson, J. A., & Coons, S. J. (2005) · Medical Care · doi:10.1097/00005650-200503000-00003 · Cited by 1,271 · see citing works
Miscellanea. Small-sample degrees of freedom with multiple imputation
Barnard, J. (1999) · Biometrika · doi:10.1093/biomet/86.4.948 · Cited by 841 · see citing works
Missing Data, Imputation, and the Bootstrap
Efron, B. (1994) · Journal of the American Statistical Association · doi:10.1080/01621459.1994.10476768 · Cited by 471 · see citing works
Missing.... presumed at random: cost‐analysis of incomplete data
Briggs, A. H., Clark, T. G., Wolstenholme, J. L., & Clarke, P. (2002) · Health Economics · doi:10.1002/hec.766 · Cited by 346 · see citing works
Use of Multiple Imputation in the Epidemiologic Literature
Klebanoff, M. A., & Cole, S. R. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn071 · Cited by 238 · see citing works
The Use of the Bootstrap Statistical Method for the Pharmacoeconomic Cost Analysis of Skewed Data
Desgagné, A., Castilloux, A., Angers, J., & LeLorier, J. (1998) · PharmacoEconomics · doi:10.2165/00019053-199813050-00002 · Cited by 180 · see citing works
Predictive mean matching imputation of semicontinuous variables
Vink, G., Frank, L. E., Pannekoek, J., & van Buuren, S. (2014) · Statistica Neerlandica · doi:10.1111/stan.12023 · Cited by 178 · see citing works
Variable selection under multiple imputation using the bootstrap in a prognostic study
Heymans, M. W., van Buuren, S., Knol, D. L., van Mechelen, W., & de Vet, H. C. W. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-33 · Cited by 172 · see citing works
van den Hout, W. B., Peul, W. C., Koes, B. W., Brand, R., Kievit, J., Thomeer, R. T. W. M., & for the Leiden-The Hague Spine Intervention Prognostic Study Group (2008) · BMJ · doi:10.1136/bmj.39583.709074.be · Cited by 135 · see citing works
Brownstone, D. M., & Valletta, R. G. (2001) · The Journal of Economic Perspectives · doi:10.1257/jep.15.4.129 · Cited by 133 · see citing works
Combining multiple imputation and meta‐analysis with individual participant data
Burgess, S., White, I. R., Resche‐Rigon, M., & Wood, A. (2013) · Statistics in Medicine · doi:10.1002/sim.5844 · Cited by 105 · see citing works
Impact of the Bootstrap on Sample Surveys
Shao, J. (2003) · Statistical Science · doi:10.1214/ss/1063994974 · Cited by 83 · see citing works
Nonparametric confidence intervals for the one- and two-sample problems.
Zhou, X.-H., & Dinh, P. (2005) · Biostatistics · doi:10.1093/biostatistics/kxi002 · Cited by 73 · see citing works
Should a Normal Imputation Model be Modified to Impute Skewed Variables?
von Hippel, P. T. (2012) · Sociological Methods & Research · doi:10.1177/0049124112464866 · Cited by 68 · see citing works
The Effect on the t Distribution of Non-Normality in the Sampled Population
Ratcliffe, J. F. (1968) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2985264 · Cited by 45 · see citing works
Comparison of imputation variance estimators
Hughes, R. A., Sterne, J. A. C., & Tilling, K. M. (2014) · Statistical Methods in Medical Research · doi:10.1177/0962280214526216 · Cited by 38 · see citing works
Briggs, A. H., Lozano‐Ortega, G., Spencer, S., Bale, G., Spencer, M. D., & Burge, P. S. (2006) · Value in Health · doi:10.1111/j.1524-4733.2006.00106.x · Cited by 30 · see citing works
Eveleigh, R., Grutters, J., Muskens, E., Oude Voshaar, R., van Weel, C., Speckens, A., & Lucassen, P. (2014) · Family Practice · doi:10.1093/fampra/cmu043 · Cited by 28 · see citing works
Multiple imputation and other resampling schemes for imputing missing observations
Srivastava, M. S., & Dolatabadi, M. (2009) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2009.06.003 · Cited by 18 · see citing works
Confounding and missing data in cost-effectiveness analysis: comparing different methods
Härkänen, T., Maljanen, T., Lindfors, O., Virtala, E. T., & Knekt, P. (2013) · Health Economics Review · doi:10.1186/2191-1991-3-8 · Cited by 18 · see citing works
(2012) · Chapman & Hall/CRC machine learning & pattern recognition/Chapman & Hall/CRC machine learning & pattern recognition series · doi:10.1201/b11826-4 · Cited by 5 · see citing works
Multiple imputation of missing blood pressure covariates in survival analysis
van Buuren, S., Boshuizen, H. C., & Knook, D. L. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990330)18:6
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Dolan P. Modeling valuations for EuroQol health states. Med Care. 1997;35(11):1095-1108.
Cited by
Sorted by citations
Correlates of protection against symptomatic and asymptomatic SARS-CoV-2 infection
Feng, S., Phillips, D. J., White, T., Sayal, H., Aley, P. K., Bibi, S., Dold, C., Fuskova, M., Gilbert, S. C., Hirsch, I., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01540-1 · Cited by 1,163 · see citing works
Handling missing data in clinical research
Heymans, M. W., & Twisk, J. W. R. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.08.016 · Cited by 399 · see citing works
Risk-based, 6-monthly and 24-monthly dental check-ups for adults: the INTERVAL three-arm RCT
Clarkson, J. E., Pitts, N. B., Goulao, B., Boyers, D., Ramsay, C. R., Floate, R., Braid, H. J., Fee, P. A., Ord, F. S., Worthington, H. V., et al. (2020) · Health Technology Assessment · doi:10.3310/hta24600 · Cited by 35 · see citing works
Bockting, C., Bosmans, G., Bergers, N., Gavan, L., Hiligsmann, M., de Beurs, D., Molenberghs, G., Wijnen, B., Lokkerbol, J., & van der Spek, N. (2024) · Trials · doi:10.1186/s13063-024-08499-7 · Cited by 1 · see citing works
“Missing cost and outcome data will be imputed using single imputation based on predictive mean matching nested in non-parametric bootstraps [70] for the ITT analysis.”
Method
- Authors
- Brand, J., van Buuren, S., le Cessie, S., & van den Hout, W.
- Affiliations
- Leiden University Medical Center (NL)
- Department of Methodology & Statistics University of Utrecht Utrecht The Netherlands
- Department of Medical Statistics Leiden University Medical Center Leiden The Netherlands
- Department of Medical Decision Making & Quality of Care Leiden University Medical Center Leiden The Netherlands
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2018-09-12
- Volume
- 38
- Issue
- 2
- Pages
- 210-220
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- PMC
- PMC6585698
- OpenAlex
- W2890400920
- Cited by
- 131
- References
- 35
- Citations by year
- 2026: 16 · 2025: 25 · 2024: 17 · 2023: 23 · 2022: 19 · 2021: 13 · 2020: 16 · 2019: 2
- FWCI
- 6.43
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Statistics in Medicine (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- TNO Repository (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Repository hosted by TU Delft Library (TU Delft) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- ZonMw (152002047)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- multiple imputation, bootstrapping, cost-effectiveness analysis, missing data, skewed distributions, confidence intervals, statistical efficiency, clinical trials
- MeSH terms
- Clinical Trials as Topic, Cost-Benefit Analysis, Data Interpretation, Statistical, Humans, Sciatica, Models, Statistical, Randomized Controlled Trials as Topic, Treatment Outcome, Health Care Costs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
