CoChat
Sign Up
Cited by 185Statistical Methods in Clinical TrialsStatistical Methods and Bayesian InferenceMeta-analysis and systematic reviews

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.

Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review, reanalysis, and simulation study

Open access
Kahan, Brennan C., Forbes, Gordon, Ali, Yunus, Jairath, Vipul, Bremner, Stephen, Harhay, Michael O., Hooper, Richard, Wright, Neil, Eldridge, Sandra M., Leyrat, Clémence·2016·Trials
DOI 10.1186/s13063-016-1571-2PMC PMC5013635
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Cluster randomised trials (CRTs) are commonly analysed using mixed-effects models or generalised estimating equations (GEEs). However, these analyses do not always perform well with the small number of clusters typical of most CRTs. They can lead to increased risk of a type I error (finding a statistically significant treatment effect when it does not exist) if appropriate corrections are not used. METHODS: We conducted a small simulation study to evaluate the impact of using small-sample corrections for mixed-effects models or GEEs in CRTs with a small number of clusters. We then reanalysed data from TRIGGER, a CRT with six clusters, to determine the effect of using an inappropriate analysis method in practice. Finally, we reviewed 100 CRTs previously identified by a search on PubMed in order to assess whether trials were using appropriate methods of analysis. Trials were classified as at risk of an increased type I error rate if they did not report using an analysis method which accounted for clustering, or if they had fewer than 40 clusters and performed an individual-level analysis without reporting the use of an appropriate small-sample correction. RESULTS: Our simulation study found that using mixed-effects models or GEEs without an appropriate correction led to inflated type I error rates, even for as many as 70 clusters. Conversely, using small-sample corrections provided correct type I error rates across all scenarios. Reanalysis of the TRIGGER trial found that inappropriate methods of analysis gave much smaller P values (P ≤ 0.01) than appropriate methods (P = 0.04-0.15). In our review, of the 99 trials that reported the number of clusters, 64 (65 %) were at risk of an increased type I error rate; 14 trials did not report using an analysis method which accounted for clustering, and 50 trials with fewer than 40 clusters performed an individual-level analysis without reporting the use of an appropriate correction. CONCLUSIONS: CRTs with a small or medium number of clusters are at risk of an inflated type I error rate unless appropriate analysis methods are used. Investigators should consider using small-sample corrections with mixed-effects models or GEEs to ensure valid results.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13063-016-1571-2

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Association between type I error rate and number of clusters for ‘appropriate’ and ‘inappropriate’ methods of analysis. This figure presents the association between the total number of clusters and the risk of an inflated type I error rate for different methods of analysis. Panels a and b show the type I error rate for methods of analysis that are ‘inappropriate’ to use with a small number of clusters. Panels c and d show the type I error rate for methods of analysis that are ‘appropriate’ to use with a small number of clusters. The mixed-effects models in panels c and d used a Satterthwaite degree-of-freedom correction. The GEE models in panels c and d used the correction proposed by Fay and Graubard. Panels a and c represent a small cluster scenario (e.g. a family). Panels b and d represent a large cluster scenario (e.g. a hospital). The intraclass correlation coefficient (ICC; a measure of how correlated participants within a cluster are) was set as 0.15 for ‘small’ clusters (panels a and c), and as 0.01 for ‘large’ clusters (panels b and d). Data were estimated using simulation (50,000 replications per scenario)
Fig. 1 Association between type I error rate and number of clusters for ‘appropriate’ and ‘inappropriate’ methods of analysis. This figure presents the association between the total number of clusters and the risk of an inflated type I error rate for different methods of analysis. Panels a and b show the type I error rate for methods of analysis that are ‘inappropriate’ to use with a small number of clusters. Panels c and d show the type I error rate for methods of analysis that are ‘appropriate’ to use with a small number of clusters. The mixed-effects models in panels c and d used a Satterthwaite degree-of-freedom correction. The GEE models in panels c and d used the correction proposed by Fay and Graubard. Panels a and c represent a small cluster scenario (e.g. a family). Panels b and d represent a large cluster scenario (e.g. a hospital). The intraclass correlation coefficient (ICC; a measure of how correlated participants within a cluster are) was set as 0.15 for ‘small’ clusters (panels a and c), and as 0.01 for ‘large’ clusters (panels b and d). Data were estimated using simulation (50,000 replications per scenario)CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 1

  1. Murray DM. Design and analysis of group-randomised trials. London: Oxford University Press; 1998.

Cited by

Sorted by citations

See all 185

Related

See all

Metadata

Authors
Kahan, B. C., Forbes, G., Ali, Y., Jairath, V., Bremner, S., Harhay, M. O., Hooper, R., Wright, N., Eldridge, S. M., & Leyrat, C.
Affiliations
  • Queen Mary University of London (GB)
  • Western University (CA)
  • Brighton and Sussex Medical School (GB)
  • University of Pennsylvania (US)
  • Inserm (FR)
Corresponding author
Kahan, Brennan C.
ORCID
Venue
Trials
Publisher
Springer Science and Business Media LLC
Published
2016-09-06
Volume
17
Issue
1
Pages
438-438
ISSN
1745-6215 (ISSN-L)
DOI
10.1186/s13063-016-1571-2
PMC
PMC5013635
OpenAlex
W2515120484
Cited by
185
References
29
Citations by year
2026: 10 · 2025: 11 · 2024: 22 · 2023: 12 · 2022: 16 · 2021: 35 · 2020: 26 · 2019: 16 · 2018: 13 · 2017: 15 · 2016: 9
FWCI
17.86
Type
journal article
Language
en
Open access
Licence
Locations
  • Trials (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
  • Sussex Research Online (University of Sussex) (submittedVersion, cc-by)
  • Figshare (submittedVersion, cc-by)
  • Research Portal (King's College London) (submittedVersion, other-oa)
  • Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
Funders
  • National Institute for Health and Care Research (NIHR-CTF-2014-03-05)
  • National Institutes of Health
  • National Heart, Lung, and Blood Institute (F31HL127947)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
type I error, small sample correction, mixed models, generalized estimating equations, small number of clusters, clustering, statistical power
MeSH terms
Computer Simulation, Data Interpretation, Statistical, Humans, Risk, Cluster Analysis, Randomized Controlled Trials as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review