Cited by 107Statistical Methods and Bayesian InferenceStatistical Methods in Clinical TrialsAdvanced Causal Inference Techniques

The DOI matches its Crossref record; 61 of the 65 references it cites were found in a registry or an index.

Using Cluster Bootstrapping to Analyze Nested Data With a Few Clusters

Open access

Huang, Francis L.·2018·Educational and Psychological Measurement

DOI 10.1177/0013164416678980PMC PMC5965657

Open this paper in CoChat — free account

Abstract

Cluster randomized trials involving participants nested within intact treatment and control groups are commonly performed in various educational, psychological, and biomedical studies. However, recruiting and retaining intact groups present various practical, financial, and logistical challenges to evaluators and often, cluster randomized trials are performed with a low number of clusters (~20 groups). Although multilevel models are often used to analyze nested data, researchers may be concerned of potentially biased results due to having only a few groups under study. Cluster bootstrapping has been suggested as an alternative procedure when analyzing clustered data though it has seen very little use in educational and psychological studies. Using a Monte Carlo simulation that varied the number of clusters, average cluster size, and intraclass correlations, we compared standard errors using cluster bootstrapping with those derived using ordinary least squares regression and multilevel models. Results indicate that cluster bootstrapping, though more computationally demanding, can be used as an alternative procedure for the analysis of clustered data when treatment effects at the group level are of primary interest. Supplementary material showing how to perform cluster bootstrapped regressions using R is also provided.

Integrity signals

Registry

Crossref · 10.1177/0013164416678980

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
lavaan : An R Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602
SPSS and SAS procedures for estimating indirect effects in simple mediation models
Preacher, K. J., & Hayes, A. F. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03206553 · Cited by 17,383
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,820
Bootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350
Untitled work
(2010) · Cited by 7,034
A Practitioner’s Guide to Cluster-Robust Inference
Colin Cameron, A., & Miller, D. L. (2015) · Journal of Human Resources · doi:10.3368/jhr.50.2.317 · Cited by 5,037
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences.
Fong, G. T., Cohen, J., & Cohen, P. R. (1985) · Journal of the American Statistical Association · doi:10.2307/2287930 · Cited by 4,610
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552
Centering predictor variables in cross-sectional multilevel models: A new look at an old issue.
Enders, C. K., & Tofighi, D. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.2.121 · Cited by 4,462
Bootstrap-Based Improvements for Inference With Clustered Errors
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008) · Review of Economics and Statistics · doi:10.1162/rest.90.3.414 · Cited by 4,417
Sufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.86 · Cited by 3,582
An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Units
Moulton, B. R. (1990) · The Review of Economics and Statistics · doi:10.2307/2109724 · Cited by 2,567
How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power
Muthén, L. K., & Muthén, B. O. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0904_8 · Cited by 2,362
Principled missing data methods for researchers
Dong, Y., & Peng, C.-Y. J. (2013) · SpringerPlus · doi:10.1186/2193-1801-2-222 · Cited by 2,208
Applied Regression Analysis and Generalized Linear Models
Fox, J. (2008) · Cited by 2,051
Multilevel Statistical Models
Goldstein, H. (2010) · Wiley series in probability and statistics · doi:10.1002/9780470973394 · Cited by 1,581
Random group effects and the precision of regression estimates
Moulton, B. R. (1986) · Journal of Econometrics · doi:10.1016/0304-4076(86)90021-7 · Cited by 1,574
A Kenward-Roger Approximation and Parametric Bootstrap Methods for Tests in Linear Mixed Models - The R Package pbkrtest
Halekoh, U., & Højsgaard, S. (2014) · Journal of Statistical Software · doi:10.18637/jss.v059.i09 · Cited by 1,324 · cited 1×
“In addition, a Kenward–Roger (Kenward & Roger, 1997) correction using the pbkrtest package (Halekoh & Højsgaard, 2014) in R was used to adjust standard errors which has shown to result in less-biased standard errors when analyzing data with a low number of clusters (B. A. Bell et al., 2014; McNeish…”
Method
Intraclass Correlation Values for Planning Group-Randomized Trials in Education
Hedges, L. V., & Hedberg, E. C. (2007) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373707299706 · Cited by 847
The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration
McNeish, D., & Stapleton, L. M. (2014) · Educational Psychology Review · doi:10.1007/s10648-014-9287-x · Cited by 726
Estimating Statistical Power and Required Sample Sizes for Organizational Research Using Multilevel Modeling
Scherbaum, C. A., & Ferreter, J. M. (2009) · Organizational Research Methods · doi:10.1177/1094428107308906 · Cited by 702
Methods matter improving causal inference in educational and social science research
Murnane, R. J., & Willett, J. B. (2011) · Cited by 620
Bootstrapping Clustered Data
Field, C., & Welsh, A. H. (2007) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2007.00593.x · Cited by 466
Pitfalls of and controversies in cluster randomization trials
Donner, A., & Klar, N. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.416 · Cited by 442
Mastering 'metrics: the path from cause to effect
(2015) · Choice Reviews Online · doi:10.5860/choice.189854 · Cited by 389
When can group level clustering be ignored? Multilevel models versus single-level models with sparse data
Clarke, P. J. (2008) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2007.060798 · Cited by 367
More Statistical and Methodological Myths and Urban Legends
(2014) · Routledge eBooks · doi:10.4324/9780203775851 · Cited by 354
Multilevel Analysis Methods
Hox, J. J., & Kreft, I. G. G. (1994) · Sociological Methods & Research · doi:10.1177/0049124194022003001 · Cited by 312
An Introduction to Multilevel Modeling Techniques
Heck, R. H., & Thomas, S. L. (2015) · Routledge eBooks · doi:10.4324/9781315746494 · Cited by 312
Alternatives to Multilevel Modeling for the Analysis of Clustered Data
Huang, F. L. (2014) · The Journal of Experimental Education · doi:10.1080/00220973.2014.952397 · Cited by 284
Self-citation
Effect of a quality-improvement intervention on end-of-life care in the intensive care unit: A randomized trial.
Curtis, J. R., Nielsen, E. L., Treece, P. D., Downey, L., Dotolo, D., Shannon, S. E., Back, A. L., Rubenfeld, G. D., & Engelberg, R. A. (2011) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201006-1004oc · Cited by 269
Creating birds of similar feathers: Leveraging similarity to improve teacher–student relationships and academic achievement.
Gehlbach, H., Brinkworth, M. E., King, A. M., Hsu, L. M., McIntyre, J., & Rogers, T. (2016) · Journal of Educational Psychology · doi:10.1037/edu0000042 · Cited by 260
Data with Hierarchical Structure: Impact of Intraclass Correlation and Sample Size on Type-I Error
Musca, S. C., Kamiejski, R., Nugier, A., Méot, A., Er-Rafiy, A., & Brauer, M. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00074 · Cited by 259
Examining the Rule of Thumb of Not Using Multilevel Modeling: The “Design Effect Smaller Than Two” Rule
Lai, M. H. C., & Kwok, O. (2014) · The Journal of Experimental Education · doi:10.1080/00220973.2014.907229 · Cited by 248
An Introduction to Bootstrap Methods with Applications to R
Chernick, M. R., & LaBudde, R. A. (2011) · ODU Digital Commons (Old Dominion University) · Cited by 242
The instructor’s face in video instruction: Evidence from two large-scale field studies.
Kizilcec, R. F., Bailenson, J. N., & Gomez, C. (2015) · Journal of Educational Psychology · doi:10.1037/edu0000013 · Cited by 236
Multilevel Modeling: A Review of Methodological Issues and Applications
Dedrick, R. F., Ferron, J. M., Hess, M. R., Hogarty, K. Y., Kromrey, J. D., Lang, T. R., Niles, J. D., & Lee, R. S. (2009) · Review of Educational Research · doi:10.3102/0034654308325581 · Cited by 217
PowerUp!: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-Experimental Design Studies
Dong, N., & Maynard, R. (2013) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2012.673143 · Cited by 206
Modeling Certainty with Clustered Data: A Comparison of Methods
Arceneaux, K., & Nickerson, D. W. (2009) · Political Analysis · doi:10.1093/pan/mpp004 · Cited by 198
How Low Can You Go?
Bell, B. A., Morgan, G. B., Schoeneberger, J. A., Kromrey, J. D., & Ferron, J. M. (2012) · Methodology · doi:10.1027/1614-2241/a000062 · Cited by 194
A Primer for Using and Understanding Weights With National Datasets
Hahs‐Vaughn, D. L. (2005) · The Journal of Experimental Education · doi:10.3200/jexe.73.3.221-248 · Cited by 194
Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000–8
Ivers, N. M., Taljaard, M., Dixon, S., Bennett, C., McRae, A., Taleban, J., Skea, Z., Brehaut, J. C., Boruch, R. F., Eccles, M. P., et al. (2011) · BMJ · doi:10.1136/bmj.d5886 · Cited by 193
Dealing With Dependence (Part I): Understanding the Effects of Clustered Data
McCoach, D. B., & Adelson, J. L. (2010) · Gifted Child Quarterly · doi:10.1177/0016986210363076 · Cited by 166
Resampling Multilevel Models
van der Leeden, R., Meijer, E., & Busing, F. M. T. A. (2007) · Springer eBooks · doi:10.1007/978-0-387-73186-5_11 · Cited by 136
Direct and Extended Vocabulary Instruction in Kindergarten: Investigating Transfer Effects
Coyne, M. D., McCoach, D. B., Loftus, S., Zipoli, R. P., Ruby, M. M., Crevecoeur, Y. C., & Kapp, S. (2010) · Journal of Research on Educational Effectiveness · doi:10.1080/19345741003592410 · Cited by 130
A comparison of bootstrap approaches for estimating uncertainty of parameters in linear mixed‐effects models
Thai, H., Mentré, F., Holford, N. H. G., Veyrat‐Follet, C., & Comets, E. (2013) · Pharmaceutical Statistics · doi:10.1002/pst.1561 · Cited by 105
Substantial risks associated with few clusters in cluster randomized and stepped wedge designs
Taljaard, M., Teerenstra, S., Ivers, N. M., & Fergusson, D. A. (2016) · Clinical Trials · doi:10.1177/1740774516634316 · Cited by 96
From the Help Desk: Some Bootstrapping Techniques
Poi, B. P. (2004) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0400400308 · Cited by 91
A Bootstrap Method for Conducting Statistical Inference with Clustered Data
Harden, J. J. (2011) · State Politics & Policy Quarterly · doi:10.1177/1532440011406233 · Cited by 90
Marketing Vegetables in Elementary School Cafeterias to Increase Uptake
Hanks, A., Just, D. R., & Brumberg, A. (2015) · PEDIATRICS · doi:10.1542/peds.2015-1720 · Cited by 60
Affect and overconfidence: A laboratory investigation.
Ifcher, J., & Zarghamee, H. (2014) · Journal of Neuroscience Psychology and Economics · doi:10.1037/npe0000022 · Cited by 51
Use of the bootstrap in analysing cost data from cluster randomised trials: some simulation results
Flynn, T. N., & Peters, T. J. (2004) · BMC Health Services Research · doi:10.1186/1472-6963-4-33 · Cited by 32
How large are the consequences of covariate imbalance in cluster randomized trials: a simulation study with a continuous outcome and a binary covariate at the cluster level
Moerbeek, M., & van Schie, S. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0182-7 · Cited by 31
Bootstrapping within the Multilevel/Hierarchical Linear Modeling Framework: A Primer for Use with SAS and SPLUS
Roberts, J. K., & Fan, X. (2004) · General Linear Model Journal · Cited by 27
Multilevel bootstrap analysis with assumptions violated
Vallejo, G., García, M. A., Fernández, P., & Livacic-Rojas, P. E. (2013) · Psicothema · doi:10.7334/psicothema2013.58 · Cited by 26
Effects of adolescent sociocognitive development on the cortisol response to social evaluation.
van den Bos, E., van Duijvenvoorde, A. C. K., & Westenberg, P. M. (2016) · Developmental Psychology · doi:10.1037/dev0000133 · Cited by 19
Are Multilevel Techniques Necessary? An Attempt at Demystification.
Kreft, I. G. G., & Yoon, B. (1994) · Cited by 10

Research works printed but not linked: 8

Bell R., 2002, Survey Methodology, 28, 169
Kish L., 1965, Survey sampling
Meijer E., 1998, Assumptions, robustness, and estimation methods in multivariate modeling, 35
Mundfrom D. J., 2001, Multiple Linear Regression Viewpoints, 27, 3
Roberts J., 2004, Multiple Linear Regression Viewpoints, 30, 23
Snijders T., 2012, Multilevel analysis
Spybrook J., 2011, Optimal design plus empirical evidence: Documentation for the “Optimal Design” software
StataCorp, 2015, Stata statistical software: Release 14

Sorted by citations

Small Sample Methods for Multilevel Modeling: A Colloquial Elucidation of REML and the Kenward-Roger Correction
McNeish, D. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1344538 · Cited by 308
“…2014), a comparison of small sample corrections (McNeish & Stapleton, 2016a), a comparison of multilevel bootstrapping and small sample corrections (Huang, 2017), comparisons of Bayesian and frequentist methods (Hox, van de Schoot, & Matthijsse, 2012), and a review of small sample methods for…”
Application of the Hierarchical Bootstrap to Multi-Level Data in Neuroscience
Saravanan, V., Berman, G. J., & Sober, S. J. (2019) · doi:10.1101/819334 · Cited by 170
“The hierarchical bootstrap [16, 17, 18, 19] is a statistical method that has been applied successfully to awide variety of clustered datasets, including census and polling results, educational and psychological data, and phylogenetic tree information [20, 21, 14].”
BackgroundMethod
Selective exposure in different political information environments – How media fragmentation and polarization shape congruent news use
Steppat, D., Castro, L., & Esser, F. (2021) · European Journal of Communication · doi:10.1177/02673231211012141 · Cited by 104
“To account for the small number of level 2 units (five countries), we follow the approach suggested by Huang (2018) and estimated cluster bootstrapped regressions with 1000 replications (clustered on countries) using the bootcov function in the rms: Regression Modelling Strategies package by…”
Method
Beyond the individual: A systematic review of the effects of unit-level demands and resources on employee productivity, health, and well-being.
Roczniewska, M. A., Smoktunowicz, E., Coo, C., Schwarz, U. V. T., Hasson, H., & Richter, A. (2021) · Journal of Occupational Health Psychology · doi:10.1037/ocp0000311 · Cited by 93
Multilevel Modeling and Ordinary Least Squares Regression: How Comparable Are They?
Huang, F. L. (2018) · The Journal of Experimental Education · doi:10.1080/00220973.2016.1277339 · Cited by 60
“…is often done with FE models, which include dummy-coded variables for dozens or even hundreds of group clusters where coefficients for each and every dummy code are not reported when the group-level variable is merely noise to be accounted for and not of substantive interest (Huang, 2016a).”
BackgroundMethodInfluentialSelf-citation

See all 107

Authors: Huang, F. L.
Affiliations: University of Missouri (US)
Corresponding author: Huang, Francis L.
ORCID: Huang, Francis L.
Venue: Educational and Psychological Measurement
Publisher: SAGE Publications
Published: 2016-11-24
Volume: 78
Issue: 2
Pages: 297-318
ISSN: 0013-1644 (ISSN-L)
1552-3888
DOI: 10.1177/0013164416678980
PMC: PMC5965657
OpenAlex: W2554437640
Cited by: 107
References: 65
Citations by year: 2026: 22 · 2025: 22 · 2024: 18 · 2023: 12 · 2022: 10 · 2021: 13 · 2020: 3 · 2019: 2 · 2018: 2 · 2017: 3
FWCI: 2.36
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Educational and Psychological Measurement (publishedVersion)
Europe PMC (PubMed Central) (publishedVersion)
PubMed Central (publishedVersion)
PubMed (publishedVersion)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords: small number of clusters, nested data, multilevel models, standard errors, intraclass correlation coefficient, cluster size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Using Cluster Bootstrapping to Analyze Nested Data With a Few Clusters

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks