CoChat
Sign Up
Cited by 32Statistical Methods in Clinical TrialsStatistical Methods and Bayesian InferenceMeta-analysis and systematic reviews

Use of the bootstrap in analysing cost data from cluster randomised trials: some simulation results

Open access
Flynn, Terry N, Peters, Tim J·2004·BMC Health Services Research
DOI 10.1186/1472-6963-4-33PMC PMC535558
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: This work has investigated under what conditions confidence intervals around the differences in mean costs from a cluster RCT are suitable for estimation using a commonly used cluster-adjusted bootstrap in preference to methods that utilise the Huber-White robust estimator of variance. The bootstrap's main advantage is in dealing with skewed data, which often characterise patient costs. However, it is insufficiently well recognised that one method of adjusting the bootstrap to deal with clustered data is only valid in large samples. In particular, the requirement that the number of clusters randomised should be large would not be satisfied in many cluster RCTs performed to date. Methods: The performances of confidence intervals for simple differences in mean costs utilising a robust (cluster-adjusted) standard error and from two cluster-adjusted bootstrap procedures were compared in terms of confidence interval coverage in a large number of simulations. Parameters varied included the intracluster correlation coefficient, the sample size and the distributions used to generate the data. Results: The bootstrap's advantage in dealing with skewed data was found to be outweighed by its poor confidence interval coverage when the number of clusters was at the level frequently found in cluster RCTs in practice. Simulations showed that confidence intervals based on robust methods of standard error estimation achieved coverage rates between 93.5% and 94.8% for a 95% nominal level whereas those for the bootstrap ranged between 86.4% and 93.8%. Conclusion: In general, 24 clusters per treatment arm is probably the minimum number for which one would even begin to consider the bootstrap in preference to traditional robust methods, for the parameter combinations investigated here. At least this number of clusters and extremely skewed data would be necessary for the bootstrap to be considered in favour of the robust method. There is a need for further investigation of more complex bootstrap procedures if economic data from cluster RCTs are to be analysed appropriately.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1472-6963-4-33

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This study supports Flynn et al.'s reasoning that we need a large number of clusters, more than 20 per group, to start considering bootstrap methods in place of more traditional robust methods like mixed models.

    “Moreover, this study supports Flynn et al.’s reasoning [23] that we need a large number of clusters, more than 20 per group, to start considering bootstrap methods in place of more traditional robust methods like mixed models.”

    From Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters (2022) · cites this paper as [23]

    The Conclusions (page unavailable) say that 24 clusters per treatment arm is probably the minimum at which to begin considering bootstrap methods over traditional robust methods, for the simulated conditions. But the Methods section, “Comparison of methods that allow for clustering” (page unavailable), identifies the comparator as a Huber–White cluster-adjusted method, not a mixed model. The conclusion also requires extremely skewed data for the bootstrap to be considered in its favour. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Flynn et al. concluded that in general, 24 clusters per treatment arm is probably the minimum number for which one would even begin to consider the bootstrap in preference to traditional robust methods, for the parameter combinations investigated.

    “Flynn et al [23] concluded that “In general, 24 clusters per treatment arm is probably the minimum number for which one would even begin to consider the bootstrap in preference to traditional robust methods, for the parameter combinations investigated here.”

    From Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters (2022) · cites this paper as [23]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Flynn, T. N., & Peters, T. J.
Affiliations
University of Bristol (GB)
Corresponding author
Flynn, Terry N
Venue
BMC Health Services Research
Publisher
Springer Science and Business Media LLC
Published
2004-11-18
Volume
4
Issue
1
Pages
33-33
ISSN
1472-6963 (ISSN-L)
DOI
10.1186/1472-6963-4-33
PMC
PMC535558
OpenAlex
W1569909498
Cited by
32
References
17
Citations by year
2024: 1 · 2023: 2 · 2022: 2 · 2021: 1 · 2020: 1 · 2018: 1 · 2017: 2 · 2016: 1 · 2014: 1 · 2013: 2 · 2012: 3
FWCI
2.03
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Health Services Research (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Medical Research Council
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
bootstrapping, cost data, confidence interval coverage, intra-cluster correlation, number of clusters, skewed data, economic evaluation
MeSH terms
Analysis of Variance, Data Interpretation, Statistical, Humans, Models, Statistical, Cluster Analysis, Confidence Intervals, Randomized Controlled Trials as Topic, Treatment Outcome, Health Care Costs, Sample Size, Statistics, Nonparametric

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review