Comparing denominator degrees of freedom approximations for the generalized linear mixed model in analyzing binary outcome in small sample cluster-randomized trials
Open accessAbstract
BACKGROUND: Small number of clusters and large variation of cluster sizes commonly exist in cluster-randomized trials (CRTs) and are often the critical factors affecting the validity and efficiency of statistical analyses. F tests are commonly used in the generalized linear mixed model (GLMM) to test intervention effects in CRTs. The most challenging issue for the approximate Wald F test is the estimation of the denominator degrees of freedom (DDF). Some DDF approximation methods have been proposed, but their small sample performances in analysing binary outcomes in CRTs with few heterogeneous clusters are not well studied. METHODS: The small sample performances of five DDF approximations for the F test are compared and contrasted under CRT frameworks with simulations. Specifically, we illustrate how the intraclass correlation (ICC), sample size, and the variation of cluster sizes affect the type I error and statistical power when different DDF approximation methods in GLMM are used to test intervention effect in CRTs with binary outcomes. The results are also illustrated using a real CRT dataset. RESULTS: Our simulation results suggest that the Between-Within method maintains the nominal type I error rates even when the total number of clusters is as low as 10 and is robust to the variation of the cluster sizes. The Residual and Containment methods have inflated type I error rates when the cluster number is small (<30) and the inflation becomes more severe with increased variation in cluster sizes. In contrast, the Satterthwaite and Kenward-Roger methods can provide tests with very conservative type I error rates when the total cluster number is small (<30) and the conservativeness becomes more severe as variation in cluster sizes increases. Our simulations also suggest that the Between-Within method is statistically more powerful than the Satterthwaite or Kenward-Roger method in analysing CRTs with heterogeneous cluster sizes, especially when the cluster number is small. CONCLUSION: We conclude that the Between-Within denominator degrees of freedom approximation method for F tests should be recommended when the GLMM is used in analysing CRTs with binary outcomes and few heterogeneous clusters, due to its type I error properties and relatively higher power.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The empirical performance of some methods for generalized linear mixed models in terms of confidence interval coverage, type I error rates, and accuracy of standard error estimates has not been reported thoroughly, especially for cRCTs with a small number of clusters per condition.
“However, the empirical performance of some of those methods in terms of confidence interval coverage, type I error rates, and accuracy of standard error estimates has not been reported thoroughly, especially for cRCTs with a small number of clusters per condition [7, 8, 13].”
From Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters (2022) · cites this paper as [8]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredIgnoring the nonindependence of observations within clusters results in misestimation of the variance components and consequently an inflated type I error rate.
“Ignoring the nonindependence of observations within clusters results in misestimation of the variance components and consequently an inflated type I error rate.”
From Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters (2022) · cites this paper as [8]
The Discussion (page unavailable) says the Residual degrees-of-freedom method does not account for within-cluster correlation and produces inflated type I error rates in cluster-randomized trials. The Results limit this finding to small numbers of clusters (fewer than 30). The paper does not say that ignoring the correlation misestimates variance components or that such misestimation causes the inflation. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550 · see citing works
Generalized linear mixed models: a practical guide for ecology and evolution
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J.-S. S. (2009) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2008.10.008 · Cited by 8,910 · see citing works
Approximate Inference in Generalized Linear Mixed Models
Breslow, N. E., & Clayton, D. G. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10594284 · Cited by 4,595 · see citing works
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · see citing works
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · see citing works
Consort 2010 statement: extension to cluster randomised trials
Campbell, M. K., Piaggio, G., Elbourne, D. R., Altman, D. G., & for the CONSORT Group (2012) · BMJ · doi:10.1136/bmj.e5661 · Cited by 3,510 · see citing works
Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberger, O. (2006) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,924 · see citing works
Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10480998 · Cited by 2,667 · see citing works
Maximum Likelihood Algorithms for Generalized Linear Mixed Models
McCulloch, C. E. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10473613 · Cited by 762 · see citing works
Design and analysis of group-randomized trials: a review of recent methodological developments
Murray, D. M., Varnell, S. P., & Blitstein, J. L. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.423 · Cited by 677 · see citing works
Pitfalls of and controversies in cluster randomization trials
Donner, A., & Klar, N. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.416 · Cited by 442 · see citing works
Campbell, M., Mollison, J., & Grimshaw, J. (2001) · Statistics in Medicine · doi:10.1002/1097-0258(20010215)20:3 · Cited by 299 · see citing works
Adequacy of approximations to distributions of test statistics in complex mixed linear models
Schaalje, G. B., McBride, J. B., & Fellingham, G. W. (2002) · Journal of Agricultural Biological and Environmental Statistics · doi:10.1198/108571102726 · Cited by 283 · see citing works
Fai, A. H.-T., & Cornelius, P. L. (1996) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659608811740 · Cited by 255 · see citing works
Developments in cluster randomized trials and Statistics in Medicine
Campbell, M. J., Donner, A. P., & Klar, N. S. (2006) · Statistics in Medicine · doi:10.1002/sim.2731 · Cited by 245 · see citing works
The Intra‐Cluster Correlation Coefficient in Cluster Randomized Trials: A Review of Definitions
Eldridge, S. M., Ukoumunne, O. C., & Carlin, J. B. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00092.x · Cited by 181 · see citing works
Intraclass correlation coefficient and outcome prevalence are associated in clustered binary data
Gulliford, M. C., Adams, G., Ukoumunne, O. C., Latinovic, R., Chinn, S. H. F., & Campbell, M. J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.08.012 · Cited by 145 · see citing works
SCHLUCHTER, M. D., & ELASHOFF, J. D. (1990) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659008811295 · Cited by 124 · see citing works
Estimation and sample size considerations for clustered binary responses
Lee, E. W., & Dubin, N. (1994) · Statistics in Medicine · doi:10.1002/sim.4780131206 · Cited by 88 · see citing works
Generalized linear and nonlinear models for correlated data: theory and applications using SAS
Vonesh, E. F. (2012) · CERN Document Server (European Organization for Nuclear Research) · Cited by 77 · see citing works
Spilke, J., Piepho, H., & Hu, X. (2005) · Journal of Agricultural Biological and Environmental Statistics · doi:10.1198/108571105x58199 · Cited by 58 · see citing works
Elston, D. A. (1998) · Biometrics · doi:10.2307/2533859 · Cited by 44 · see citing works
Analysis of messy longitudinal data from a randomized clinical trial
Qian, W., Parmar, M., Sambrook, R., Fayers, P. M., Girling, D. J., & Stephens, R. J. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001015)19:19 · Cited by 36 · see citing works
Bellamy, S. L., Li, Y., Lin, X., et al. (2005) · UTS ePRESS (University of Technology Sydney) · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Cluster randomized trials with a small number of clusters: which analyses should be used?
Leyrat, C., Morgan, K. E., Leurent, B., & Kahan, B. C. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyx169 · Cited by 244 · see citing works
Review of recent methodological developments in group-randomized trials: part 2—analysis
Turner, E. L., Prague, M., Gallis, J. A., Li, F., & Murray, D. M. (2017) · American Journal of Public Health · doi:10.2105/ajph.2017.303707 · Cited by 195 · see citing works
Kahan, B. C., Forbes, G., Ali, Y., Jairath, V., Bremner, S., Harhay, M. O., Hooper, R., Wright, N., Eldridge, S. M., & Leyrat, C. (2016) · Trials · doi:10.1186/s13063-016-1571-2 · Cited by 185 · see citing works
Yan, L. L., Gong, E., Gu, W., Turner, E. L., Gallis, J. A., Zhou, Y., Li, Z., McCormack, K. E., Xu, L.-Q., Bettger, J. P., et al. (2021) · PLOS Medicine · doi:10.1371/journal.pmed.1003582 · Cited by 98 · see citing works
Bennion, K. A., Tate, D., Muñoz‐Christian, K., & Phelan, S. (2020) · Obesity · doi:10.1002/oby.22921 · Cited by 18 · see citing works
Aifah, A. A., Hade, E. M., Colvin, C., Henry, D., Mishra, S., Rakhra, A., Onakomaiya, D., Ekanem, A., Shedul, G., Bansal, G. P., et al. (2023) · Implementation Science · doi:10.1186/s13012-023-01272-5 · Cited by 11 · see citing works
Golzarri-Arroyo, L., Dickinson, S. L., Jamshidi-Naeini, Y., Zoh, R. S., Brown, A. W., Owora, A. H., Li, P., Oakes, J. M., & Allison, D. B. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106654 · Cited by 9 · see citing works
Self-citation
- Authors
- Li, P., & Redden, D. T.
- Affiliations
- University of Alabama at Birmingham (US)
- Corresponding author
- Li, Peng
- Venue
- BMC Medical Research Methodology
- Published
- 2015-04-23
- Volume
- 15
- Issue
- 1
- Pages
- 38-38
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC4458010
- OpenAlex
- W2022958256
- Cited by
- 142
- References
- 24
- Citations by year
- 2026: 11 · 2025: 16 · 2024: 8 · 2023: 13 · 2022: 18 · 2021: 25 · 2020: 14 · 2019: 11 · 2018: 11 · 2017: 11 · 2016: 4
- FWCI
- 7.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (UL1TR000165, UL1TR001417, TR000165)
- National Heart, Lung, and Blood Institute (T32HL079888)
- National Institute of Arthritis and Musculoskeletal and Skin Diseases (P60AR048095, P60AR064172)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- generalized linear mixed models, binary outcomes, small sample inference, type I error, statistical power
- MeSH terms
- Algorithms, Computer Simulation, Humans, Reproducibility of Results, Linear Models, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
