Using cluster bootstrapping to analyze nested data with a few clusters

Huang, Francis L.

doi:10.1177/0013164416678980

Cited by 107Statistical Methods and Bayesian InferenceStatistical Methods in Clinical TrialsAdvanced Causal Inference Techniques

Using cluster bootstrapping to analyze nested data with a few clusters

Open access

Huang, Francis L.·2018·Educational and Psychological Measurement

DOI 10.1177/0013164416678980PMC PMC5965657

Open this paper in CoChat — free account

Abstract

Cluster randomized trials involving participants nested within intact treatment and control groups are commonly performed in various educational, psychological, and biomedical studies. However, recruiting and retaining intact groups present various practical, financial, and logistical challenges to evaluators and often, cluster randomized trials are performed with a low number of clusters (~20 groups). Although multilevel models are often used to analyze nested data, researchers may be concerned of potentially biased results due to having only a few groups under study. Cluster bootstrapping has been suggested as an alternative procedure when analyzing clustered data though it has seen very little use in educational and psychological studies. Using a Monte Carlo simulation that varied the number of clusters, average cluster size, and intraclass correlations, we compared standard errors using cluster bootstrapping with those derived using ordinary least squares regression and multilevel models. Results indicate that cluster bootstrapping, though more computationally demanding, can be used as an alternative procedure for the analysis of clustered data when treatment effects at the group level are of primary interest. Supplementary material showing how to perform cluster bootstrapped regressions using R is also provided.

Integrity signals

Registry

Crossref · 10.1177/0013164416678980

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Huang showed that a version of a nonparametric bootstrap (cluster bootstrapping) may work well for cRCT data for studies with normally distributed data and designs with 10 or more clusters per treatment group.
“For small numbers of clusters in particular, Huang [26] showed that a version of a nonparametric bootstrap (cluster bootstrapping) may work well for cRCT data for studies with normally distributed data and designs with 10 or more clusters per treatment group.”
From Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters (2022) · cites this paper as [26]
This study confirms Huang et al.'s observation that using bootstrap methods for cRCT should have at least 10 clusters per group to have a valid type I error rate.
“We have confirmed Huang et al.’s observation [26] that using bootstrap methods for cRCT should have at least 10 clusters per group to have a valid type I error rate.”
From Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters (2022) · cites this paper as [26]

R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
lavaan : An R Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955 · see citing works
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602 · see citing works
SPSS and SAS procedures for estimating indirect effects in simple mediation models
Preacher, K. J., & Hayes, A. F. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03206553 · Cited by 17,383 · see citing works
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,820 · see citing works
Bootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350 · see citing works
Untitled work
(2010) · Cited by 7,034 · see citing works
A Practitioner’s Guide to Cluster-Robust Inference
Colin Cameron, A., & Miller, D. L. (2015) · Journal of Human Resources · doi:10.3368/jhr.50.2.317 · Cited by 5,037 · see citing works
Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences.
Fong, G. T., Cohen, J., & Cohen, P. R. (1985) · Journal of the American Statistical Association · doi:10.2307/2287930 · Cited by 4,608 · see citing works
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · see citing works
Centering predictor variables in cross-sectional multilevel models: A new look at an old issue.
Enders, C. K., & Tofighi, D. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.2.121 · Cited by 4,462 · see citing works
Bootstrap-Based Improvements for Inference With Clustered Errors
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008) · Review of Economics and Statistics · doi:10.1162/rest.90.3.414 · Cited by 4,417 · see citing works
Sufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.86 · Cited by 3,582 · see citing works
An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Units
Moulton, B. R. (1990) · The Review of Economics and Statistics · doi:10.2307/2109724 · Cited by 2,640 · see citing works
How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power
Muthén, L. K., & Muthén, B. O. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0904_8 · Cited by 2,362 · see citing works
Principled missing data methods for researchers
Dong, Y., & Peng, C.-Y. J. (2013) · SpringerPlus · doi:10.1186/2193-1801-2-222 · Cited by 2,208 · see citing works
Applied Regression Analysis and Generalized Linear Models
Fox, J. (2008) · Cited by 2,051 · see citing works
Random group effects and the precision of regression estimates
Moulton, B. R. (1986) · Journal of Econometrics · doi:10.1016/0304-4076(86)90021-7 · Cited by 1,664 · see citing works
Multilevel Statistical Models
Goldstein, H. (2010) · Wiley series in probability and statistics · doi:10.1002/9780470973394 · Cited by 1,581 · see citing works
A Kenward-Roger Approximation and Parametric Bootstrap Methods for Tests in Linear Mixed Models - The R Package pbkrtest
Halekoh, U., & Højsgaard, S. (2014) · Journal of Statistical Software · doi:10.18637/jss.v059.i09 · Cited by 1,324 · see citing works
Intraclass Correlation Values for Planning Group-Randomized Trials in Education
Hedges, L. V., & Hedberg, E. C. (2007) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373707299706 · Cited by 847 · see citing works
The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration
McNeish, D., & Stapleton, L. M. (2014) · Educational Psychology Review · doi:10.1007/s10648-014-9287-x · Cited by 726 · see citing works
Estimating Statistical Power and Required Sample Sizes for Organizational Research Using Multilevel Modeling
Scherbaum, C. A., & Ferreter, J. M. (2009) · Organizational Research Methods · doi:10.1177/1094428107308906 · Cited by 702 · see citing works
Methods matter improving causal inference in educational and social science research
Murnane, R. J., & Willett, J. B. (2011) · Cited by 620 · see citing works
Bootstrapping Clustered Data
Field, C., & Welsh, A. H. (2007) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2007.00593.x · Cited by 466 · see citing works
Pitfalls of and controversies in cluster randomization trials
Donner, A., & Klar, N. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.416 · Cited by 442 · see citing works
Mastering 'metrics: the path from cause to effect
(2015) · Choice Reviews Online · doi:10.5860/choice.189854 · Cited by 389 · see citing works
When can group level clustering be ignored? Multilevel models versus single-level models with sparse data
Clarke, P. J. (2008) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2007.060798 · Cited by 367 · see citing works
More Statistical and Methodological Myths and Urban Legends
(2014) · Routledge eBooks · doi:10.4324/9780203775851 · Cited by 350 · see citing works
Multilevel Analysis Methods
Hox, J. J., & Kreft, I. G. G. (1994) · Sociological Methods & Research · doi:10.1177/0049124194022003001 · Cited by 316 · see citing works
An Introduction to Multilevel Modeling Techniques
Heck, R. H., & Thomas, S. L. (2015) · Routledge eBooks · doi:10.4324/9781315746494 · Cited by 312 · see citing works
Alternatives to Multilevel Modeling for the Analysis of Clustered Data
Huang, F. L. (2014) · The Journal of Experimental Education · doi:10.1080/00220973.2014.952397 · Cited by 284 · see citing works
Self-citation
Effect of a quality-improvement intervention on end-of-life care in the intensive care unit: A randomized trial.
Curtis, J. R., Nielsen, E. L., Treece, P. D., Downey, L., Dotolo, D., Shannon, S. E., Back, A. L., Rubenfeld, G. D., & Engelberg, R. A. (2011) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201006-1004oc · Cited by 269 · see citing works
Data with Hierarchical Structure: Impact of Intraclass Correlation and Sample Size on Type-I Error
Musca, S. C., Kamiejski, R., Nugier, A., Méot, A., Er-Rafiy, A., & Brauer, M. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00074 · Cited by 259 · see citing works
Examining the Rule of Thumb of Not Using Multilevel Modeling: The “Design Effect Smaller Than Two” Rule
Lai, M. H. C., & Kwok, O. (2014) · The Journal of Experimental Education · doi:10.1080/00220973.2014.907229 · Cited by 248 · see citing works
Creating birds of similar feathers: Leveraging similarity to improve teacher–student relationships and academic achievement.
Gehlbach, H., Brinkworth, M. E., King, A. M., Hsu, L. M., McIntyre, J., & Rogers, T. (2016) · Journal of Educational Psychology · doi:10.1037/edu0000042 · Cited by 237 · see citing works
An Introduction to Bootstrap Methods with Applications to R
Chernick, M. R., & LaBudde, R. A. (2011) · ODU Digital Commons (Old Dominion University) · Cited by 236 · see citing works
The instructor’s face in video instruction: Evidence from two large-scale field studies.
Kizilcec, R. F., Bailenson, J. N., & Gomez, C. (2015) · Journal of Educational Psychology · doi:10.1037/edu0000013 · Cited by 220 · see citing works
Multilevel Modeling: A Review of Methodological Issues and Applications
Dedrick, R. F., Ferron, J. M., Hess, M. R., Hogarty, K. Y., Kromrey, J. D., Lang, T. R., Niles, J. D., & Lee, R. S. (2009) · Review of Educational Research · doi:10.3102/0034654308325581 · Cited by 217 · see citing works
PowerUp!: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-Experimental Design Studies
Dong, N., & Maynard, R. (2013) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2012.673143 · Cited by 206 · see citing works
Modeling Certainty with Clustered Data: A Comparison of Methods
Arceneaux, K., & Nickerson, D. W. (2009) · Political Analysis · doi:10.1093/pan/mpp004 · Cited by 198 · see citing works
How Low Can You Go?
Bell, B. A., Morgan, G. B., Schoeneberger, J. A., Kromrey, J. D., & Ferron, J. M. (2012) · Methodology · doi:10.1027/1614-2241/a000062 · Cited by 194 · see citing works
A Primer for Using and Understanding Weights With National Datasets
Hahs‐Vaughn, D. L. (2005) · The Journal of Experimental Education · doi:10.3200/jexe.73.3.221-248 · Cited by 194 · see citing works
Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000–8
Ivers, N. M., Taljaard, M., Dixon, S., Bennett, C., McRae, A., Taleban, J., Skea, Z., Brehaut, J. C., Boruch, R. F., Eccles, M. P., et al. (2011) · BMJ · doi:10.1136/bmj.d5886 · Cited by 193 · see citing works
Dealing With Dependence (Part I): Understanding the Effects of Clustered Data
McCoach, D. B., & Adelson, J. L. (2010) · Gifted Child Quarterly · doi:10.1177/0016986210363076 · Cited by 162 · see citing works
Resampling Multilevel Models
van der Leeden, R., Meijer, E., & Busing, F. M. T. A. (2007) · Springer eBooks · doi:10.1007/978-0-387-73186-5_11 · Cited by 130 · see citing works
Direct and Extended Vocabulary Instruction in Kindergarten: Investigating Transfer Effects
Coyne, M. D., McCoach, D. B., Loftus, S., Zipoli, R. P., Ruby, M. M., Crevecoeur, Y. C., & Kapp, S. (2010) · Journal of Research on Educational Effectiveness · doi:10.1080/19345741003592410 · Cited by 130 · see citing works
A comparison of bootstrap approaches for estimating uncertainty of parameters in linear mixed‐effects models
Thai, H., Mentré, F., Holford, N. H. G., Veyrat‐Follet, C., & Comets, E. (2013) · Pharmaceutical Statistics · doi:10.1002/pst.1561 · Cited by 105 · see citing works
Substantial risks associated with few clusters in cluster randomized and stepped wedge designs
Taljaard, M., Teerenstra, S., Ivers, N. M., & Fergusson, D. A. (2016) · Clinical Trials · doi:10.1177/1740774516634316 · Cited by 96 · see citing works
From the Help Desk: Some Bootstrapping Techniques
Poi, B. P. (2004) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0400400308 · Cited by 91 · see citing works
A Bootstrap Method for Conducting Statistical Inference with Clustered Data
Harden, J. J. (2011) · State Politics & Policy Quarterly · doi:10.1177/1532440011406233 · Cited by 90 · see citing works
Marketing Vegetables in Elementary School Cafeterias to Increase Uptake
Hanks, A., Just, D. R., & Brumberg, A. (2016) · PEDIATRICS · doi:10.1542/peds.2015-1720 · Cited by 60 · see citing works
Affect and overconfidence: A laboratory investigation.
Ifcher, J., & Zarghamee, H. (2014) · Journal of Neuroscience Psychology and Economics · doi:10.1037/npe0000022 · Cited by 51 · see citing works
Use of the bootstrap in analysing cost data from cluster randomised trials: some simulation results
Flynn, T. N., & Peters, T. J. (2004) · BMC Health Services Research · doi:10.1186/1472-6963-4-33 · Cited by 32 · see citing works
How large are the consequences of covariate imbalance in cluster randomized trials: a simulation study with a continuous outcome and a binary covariate at the cluster level
Moerbeek, M., & van Schie, S. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0182-7 · Cited by 31 · see citing works
Bootstrapping within the Multilevel/Hierarchical Linear Modeling Framework: A Primer for Use with SAS and SPLUS
Roberts, J. K., & Fan, X. (2004) · General Linear Model Journal · Cited by 27 · see citing works
Multilevel bootstrap analysis with assumptions violated
Vallejo, G., García, M. A., Fernández, P., & Livacic-Rojas, P. E. (2013) · Psicothema · doi:10.7334/psicothema2013.58 · Cited by 26 · see citing works
Effects of adolescent sociocognitive development on the cortisol response to social evaluation.
van den Bos, E., van Duijvenvoorde, A. C. K., & Westenberg, P. M. (2016) · Developmental Psychology · doi:10.1037/dev0000133 · Cited by 16 · see citing works
Are Multilevel Techniques Necessary? An Attempt at Demystification.
Kreft, I. G. G., & Yoon, B. (1994) · Cited by 10 · see citing works

Flagged first, then by citations

4 of the 65 references couldn’t be matched to a known paper.

Research works printed but not linked: 8

Bell R., 2002, Survey Methodology, 28, 169
Kish L., 1965, Survey sampling
Meijer E., 1998, Assumptions, robustness, and estimation methods in multivariate modeling, 35
Mundfrom D. J., 2001, Multiple Linear Regression Viewpoints, 27, 3
Roberts J., 2004, Multiple Linear Regression Viewpoints, 30, 23
Snijders T., 2012, Multilevel analysis
Spybrook J., 2011, Optimal design plus empirical evidence: Documentation for the “Optimal Design” software
StataCorp, 2015, Stata statistical software: Release 14

Sorted by citations

Small Sample Methods for Multilevel Modeling: A Colloquial Elucidation of REML and the Kenward-Roger Correction
McNeish, D. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1344538 · Cited by 279 · see citing works
“…2014), a comparison of small sample corrections (McNeish & Stapleton, 2016a), a comparison of multilevel bootstrapping and small sample corrections (Huang, 2017), comparisons of Bayesian and frequentist methods (Hox, van de Schoot, & Matthijsse, 2012), and a review of small sample methods for…”
Application of the Hierarchical Bootstrap to Multi-Level Data in Neuroscience
Saravanan, V., Berman, G. J., & Sober, S. J. (2019) · doi:10.1101/819334 · Cited by 170 · see citing works
“The hierarchical bootstrap [16, 17, 18, 19] is a statistical method that has been applied successfully to awide variety of clustered datasets, including census and polling results, educational and psychological data, and phylogenetic tree information [20, 21, 14].”
BackgroundMethod
Selective exposure in different political information environments – How media fragmentation and polarization shape congruent news use
Steppat, D., Castro, L., & Esser, F. (2021) · European Journal of Communication · doi:10.1177/02673231211012141 · Cited by 84 · see citing works
“To account for the small number of level 2 units (five countries), we follow the approach suggested by Huang (2018) and estimated cluster bootstrapped regressions with 1000 replications (clustered on countries) using the bootcov function in the rms: Regression Modelling Strategies package by…”
Method
Beyond the individual: A systematic review of the effects of unit-level demands and resources on employee productivity, health, and well-being.
Roczniewska, M. A., Smoktunowicz, E., Coo, C., Schwarz, U. V. T., Hasson, H., & Richter, A. (2021) · Journal of Occupational Health Psychology · doi:10.1037/ocp0000311 · Cited by 73 · see citing works
Multilevel Modeling and Ordinary Least Squares Regression: How Comparable Are They?
Huang, F. L. (2017) · The Journal of Experimental Education · doi:10.1080/00220973.2016.1277339 · Cited by 69 · see citing works
“…is often done with FE models, which include dummy-coded variables for dozens or even hundreds of group clusters where coefficients for each and every dummy code are not reported when the group-level variable is merely noise to be accounted for and not of substantive interest (Huang, 2016a).”
BackgroundMethodInfluentialSelf-citation
Analyzing Cross-Sectionally Clustered Data Using Generalized Estimating Equations
Huang, F. L. (2021) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986211017480 · Cited by 67 · see citing works
“Although OLS and MLM coefficients are often similar—they are not always the same and can even be in the opposite direction (Huang, 2018a).”
BackgroundMethodInfluentialSelf-citation
Using cluster-robust standard errors when analyzing group-randomized trials with few clusters
Huang, F. L., & Li, X. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01627-0 · Cited by 71 · see citing works
“We include the use of an RI-MLM as this is a common approach used when investigating treatment effects using a CRT, as coefficients and standard errors are relatively unbiased even with as few as 20 clusters (Huang, 2018a).”
BackgroundMethodSelf-citation
Comparative Performance of Contrast-enhanced Mammography, Abbreviated Breast MRI, and Standard Breast MRI for Breast Cancer Screening
Lawson, M. B., Partridge, S. C., Hippe, D. S., et al. (2023) · Radiology · doi:10.1148/radiol.230576 · Cited by 61 · see citing works
What News Users Perceive as ‘Alternative Media’ Varies between Countries: How Media Fragmentation and Polarization Matter
Steppat, D., Castro, L., & Esser, F. (2021) · Digital Journalism · doi:10.1080/21670811.2021.1939747 · Cited by 55 · see citing works
Racial/Ethnic Parity in Disciplinary Consequences Using Student Threat Assessment
Cornell, D. G., Maeng, J. L., Huang, F. L., Shukla, K. D., & Konold, T. R. (2018) · School Psychology Review · doi:10.17105/spr-2017-0030.v47-2 · Cited by 54 · see citing works
“V47-2 In addition, to avoid making any distributional assumptions of the risk ratios, as the distributions of risk ratios across schools are likely skewed, bootstrapped 95% confidence intervals for the average risk ratios were also computed (see Huang, 2016).”
Self-citation
Challenging Conventional Wisdom for Multivariate Statistical Models With Small Samples
McNeish, D. M. (2017) · Review of Educational Research · doi:10.3102/0034654317727727 · Cited by 47 · see citing works
“Though this is only one example and methodologically advanced readers will be quick to note that the results do not generalize as they would if obtained via a comprehensive simulation, these data demonstrate consistent patterns that have been found previously in such comprehensive simulations (Browne & Draper, 2006; d…”
MethodResultInfluential
A scoping review of statistical methods for trial‐based economic evaluations: The current state of play
Alili, M. E., van Dongen, J. M., Eßer, J., Heymans, M. W., van Tulder, M. W., & Bosmans, J. E. (2022) · Health Economics · doi:10.1002/hec.4603 · Cited by 41 · see citing works
Bootstrap Confidence Intervals for Multilevel Standardized Effect Size
Lai, M. H. C. (2020) · Multivariate Behavioral Research · doi:10.1080/00273171.2020.1746902 · Cited by 37 · see citing works
Digital capability, digital learning, and sustainable behaviour among university students in Taiwan: A comparison design of integrated mediation-moderation models
Liu, C.-H., Horng, J.-S., Chou, S.-F., Yu, T.-Y., Lee, M.-T., & Lapuz, M. C. B. (2023) · The International Journal of Management Education · doi:10.1016/j.ijme.2023.100835 · Cited by 35 · see citing works
Cities and regions tackle climate change mitigation but often focus on less effective solutions
Farr, K. B., Song, K., Yeo, Z. Y., Johnson, E., & Hsu, A. (2023) · Communications Earth & Environment · doi:10.1038/s43247-023-01108-6 · Cited by 35 · see citing works
Circulating Tumor DNA Assay Detects Merkel Cell Carcinoma Recurrence, Disease Progression, and Minimal Residual Disease: Surveillance and Prognostic Implications
Akaike, T., Thakuria, M., Silk, A. W., Hippe, D. S., Park, S. Y., So, N. A., Maloney, N. J., Gunnell, L., Eschholz, A., Kim, E., et al. (2024) · Journal of Clinical Oncology · doi:10.1200/jco.23.02054 · Cited by 37 · see citing works
Can the Academic and Experiential Study of Flourishing Improve Flourishing in College Students? A Multi-university Study
Hirshberg, M. J., Colaianne, B. A., Greenberg, M. T., Inkelas, K. K., Davidson, R. J., Germano, D., Dunne, J. D., & Roeser, R. W. (2022) · Mindfulness · doi:10.1007/s12671-022-01952-1 · Cited by 41 · see citing works
Enhancing the Capabilities of Continuous Glucose Monitoring With a Predictive App
Herrero, P., Andorrà, M., Babion, N., Bos, H., Koehler, M., Klopfenstein, Y., Leppäaho, E., Lustenberger, P., Peak, A., Ringemann, C., et al. (2024) · Journal of Diabetes Science and Technology · doi:10.1177/19322968241267818 · Cited by 31 · see citing works
News Media Performance Evaluated by National Audiences: How Media Environments and User Preferences Matter
Steppat, D., Castro, L., & Esser, F. (2020) · Media and Communication · doi:10.17645/mac.v8i3.3091 · Cited by 26 · see citing works
Assessing Wolbachia-mediated sterility for dengue control: emulation of a cluster-randomized target trial in Singapore
Lim, J. T., Mailepessov, D., Chong, C., et al. (2024) · Journal of Travel Medicine · doi:10.1093/jtm/taae103 · Cited by 26 · see citing works
“17 The constrained randomization procedure also helped to prevent chance imbalance in baseline dengue risk due to the small number of intervention ( n =4) locations considered (see Supplementary information).”
Background
Care partner–assisted intervention to improve oral health for older adults with cognitive impairment: A feasibility study
Wu, B., Anderson, R. A., Pei, Y., Xu, H., Nye, K. E., Poole, P., Bunn, M., Downey, C. L., & Plassman, B. L. (2021) · Gerodontology · doi:10.1111/ger.12528 · Cited by 24 · see citing works
Tutorial on Biostatistics: Receiver-Operating Characteristic (ROC) Analysis for Correlated Eye Data
Ying, G., Maguire, M. G., Glynn, R. J., et al. (2021) · Ophthalmic Epidemiology · doi:10.1080/09286586.2021.1921226 · Cited by 22 · see citing works
“Although guidelines have been suggested as to the optimal number of bootstrap replications, 1,000 replications is generally considered acceptable for standard error estimates.(16) When bootstrapping the AUC from a multivariable logistic regression model (i.”
Method
Self-directed learning in massive open online courses and its application at the workplace: Does employer support matter?
Hamori, M. (2023) · Journal of business research · doi:10.1016/j.jbusres.2022.113590 · Cited by 21 · see citing works
Analyzing nested experimental designs—A user-friendly resampling method to determine experimental significance
Kulkarni, R. U., Wang, C. L., & Bertozzi, C. R. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010061 · Cited by 22 · see citing works
Diagnostic Performance of Point-of-Care Apparent Diffusion Coefficient Measures to Reduce Biopsy in Breast Lesions at MRI: Clinical Validation
Youn, I., Biswas, D., Hippe, D. S., et al. (2024) · Radiology · doi:10.1148/radiol.232313 · Cited by 19 · see citing works
Quantitative T2 ‐mapping magnetic resonance imaging for assessment of muscle motor unit recruitment patterns
Argentieri, E. C., Tan, E. T., Whang, J. S., et al. (2021) · Muscle & Nerve · doi:10.1002/mus.27186 · Cited by 18 · see citing works
“resamples to account for the dependency from multiple muscles per subject.(23) Statistical significance of the differences in c indices is implied by bootstrapped 95% CIs that exclude the null value of 0.”
Background
Riding smooth: A cost-benefit assessment of surface quality on Copenhagen’s bicycle network
Argyros, D., Jensen, A. F., Rich, J., et al. (2024) · Sustainable Cities and Society · doi:10.1016/j.scs.2024.105473 · Cited by 18 · see citing works
Staffing effectiveness across countries: An institutional perspective
Knappert, L., Peretz, H., Aycan, Z., & Budhwar, P. S. (2021) · Human Resource Management Journal · doi:10.1111/1748-8583.12411 · Cited by 16 · see citing works
Timing of repeat epinephrine to inform paediatric anaphylaxis observation periods: a retrospective cohort study
Dribin, T. E., Sampson, H. A., Zhang, Y., et al. (2025) · The Lancet Child & Adolescent Health · doi:10.1016/s2352-4642(25)00139-7 · Cited by 12 · see citing works
Predicting Glaucoma Surgical Outcomes Using Neural Networks and Machine Learning on Electronic Health Records
Barry, S., & Wang, S. Y. (2024) · Translational Vision Science & Technology · doi:10.1167/tvst.13.6.15 · Cited by 12 · see citing works
“Metrics and their confidence intervals were derived using a clustered bootstrapping approach 11,12 to account for the nonindependence within the test set resulting from rare instances where patients underwent multiple glaucoma surgeries.”
Adult‐like neural representation of species‐specific songs in the auditory forebrain of zebra finch nestlings
Schroeder, K. M., & Remage‐Healey, L. (2020) · Developmental Neurobiology · doi:10.1002/dneu.22802 · Cited by 12 · see citing works
“, Student's t-tests) but retains statistical power with low sample sizes or small effect sizes (Huang, 2018).”
BackgroundMethod
Relationship between systolic blood pressure and mortality in older vs younger trauma patients – a retrospective multicentre observational study
Benhamed, A., Batomen, B., Boucher, V., et al. (2023) · BMC Emergency Medicine · doi:10.1186/s12873-023-00863-1 · Cited by 11 · see citing works
“Knot locations were identified heuristically using Harrel’s Regression Modelling Strategies, and 95% CI were obtained using a clustered bootstrap [23].”
Method
Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters
Golzarri-Arroyo, L., Dickinson, S. L., Jamshidi-Naeini, Y., Zoh, R. S., Brown, A. W., Owora, A. H., Li, P., Oakes, J. M., & Allison, D. B. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106654 · Cited by 9 · see citing works
Management and Prognosis of Cardiac Metastatic Merkel Cell Carcinoma: A Case–Control Study and Literature Review
Akaike, T., Cahill, K., Akaike, G., et al. (2022) · Cancers · doi:10.3390/cancers14235914 · Cited by 10 · see citing works
“The non-parametric bootstrap was used to calculate 95% confidence intervals (CIs) for the cardiac and non-cardiac mMCC OS curves [34].”
Method
Effect of an Integrated School Garden and Home Garden Intervention on Anemia Among School-Aged Children in Nepal: Evidence From a Cluster Randomised Controlled Trial
Baliki, G., Weiffen, D., Schreinemachers, P., Shrestha, A., Shrestha, R. M., Schreiner, M., & Brück, T. (2023) · Food and Nutrition Bulletin · doi:10.1177/03795721231194124 · Cited by 9 · see citing works
“47 Again, we control for household-specific characteristics.”
Single- and Multilevel Perspectives on Covariate Selection in Randomized Intervention Studies on Student Achievement
Stallasch, S. E., Lüdtke, O., Artelt, C., Hedges, L. V., & Brunner, M. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09898-7 · Cited by 9 · see citing works
“We drew 1000 samples (Huang, 2018, p. 303; Schomaker & Heumann, 2018).”
Correlation of Optical Coherence Tomography Angiography Characteristics with Visual Function to Define Vision-Threatening Diabetic Macular Ischemia
Tsai, W.-S., Thottarath, S., Gurudas, S., et al. (2022) · Diagnostics · doi:10.3390/diagnostics12051050 · Cited by 9 · see citing works
“Third, the statistical methods employed in this study, namely the generalized estimating equations (GEE) model and bias-corrected bootstrapping strategy, corrected the inter-eye correlation [22,23].”
Method
A comparison of automatic and manual measures of turn-taking in monolingual and bilingual contexts
Ramírez, N. F., Hippe, D. S., Braverman, A., Weiss, Y., & Kuhl, P. K. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02127-z · Cited by 9 · see citing works
“Confidence intervals (CIs) and p -values were calculated for the regression coefficients using the nonparametric bootstrap with resampling done by participant rather than by segment (Huang, 2018).”
Baseline characteristics of eyes with early residual fluid post loading phase of aflibercept therapy in neovascular AMD: PRECISE study report 3
Chandra, S., Gurudas, S., Pearce, I., McKibbin, M., Kotagiri, A. K., Menon, G., Burton, B., Talks, J. S., Grabowska, A., Ghanchi, F. D., et al. (2023) · Eye · doi:10.1038/s41433-023-02886-1 · Cited by 8 · see citing works
“…for CST and IRF ± SRF using clustered bootstrap with 1000 replicates to estimate the bias corrected 95% confidence intervals for the area under the curve (AUC), and optimal thresholds for CST were selected based on maximising Youden’s index, defined as sensitivity + specificity – 1 [12, 13].”
Method
Deep learning-based image quality assessment for optical coherence tomography macular scans: a multicentre study
Tang, Z., Wang, X., Ran, A. R., et al. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-323871 · Cited by 8 · see citing works
Comparison of model- and design-based approaches to detect the treatment effect and covariate by treatment interactions in three-level models for multisite cluster-randomized trials
Aydın, B., Algina, J. J., & Leite, W. L. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1080-1 · Cited by 8 · see citing works
“A relatively recent approach, cluster bootstrapping, has been shown to produce results similar to those of multilevel modeling (MLM), but the results are computationally demanding, especially within a Monte Carlo simulation study (Huang, 2016).”
Result
Clientelism and ideological competition: the impact on ideological overlapping
Barreda, M., Felipe, P. O., & Rodríguez, L. M. R. (2023) · Journal of Elections Public Opinion and Parties · doi:10.1080/17457289.2023.2190120 · Cited by 7 · see citing works
“13 Bootstrapping provides di ﬀ erent ways of calculating the con ﬁ dence intervals for the parameters obtained in estimated models (Huang 2018).”
Method
The performance of small sample correction methods for controlling type I error when analyzing parallel cluster randomized trials: a systematic review of simulation studies
Hemming, K., Thompson, J. A., Kristunas, C., Forbes, A., Watson, S., Bowden, R. A., McKenzie, J. E., & Kasza, J. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111838 · Cited by 6 · see citing works
Prospective Validation of an Automated Hybrid Multidimensional MRI Tool for Prostate Cancer Detection Using Targeted Biopsy: Comparison with PI-RADS-based Assessment
Chatterjee, A., Yousuf, A. N., Engelmann, R. M., et al. (2025) · Radiology Imaging Cancer · doi:10.1148/rycan.240156 · Cited by 6 · see citing works
Pandemic Social Distancing and Declines in Nasopharyngeal Carriage of Pneumococcus and Related Antimicrobial-Resistant Genes: Evidence From Household-Based Cohort Studies in Lima, Peru
Charnogursky, C. E., Gil, A. I., Ecker, L., et al. (2025) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofaf157 · Cited by 6 · see citing works
“The bootstrap method for clustered data was used to construct the 95% CIs based on 2.5% and 97.5% quantiles from 5000 bootstrap samples [26].”
Influential
Antimicrobial Stewardship with and without Infectious Diseases Specialist Services to Improve Quality-of-Care in Secondary and Tertiary Care Hospitals in Germany: Study Protocol of the ID ROLL OUT Study
Wimmesberger, N., Allen, R., Fink, G., Först, G., Kern, W. V., Farin‐Glattacker, E., Rieg, S., the ID ROLL OUT Study group, Solzbach, U., Friedrich, H., et al. (2021) · Infectious Diseases and Therapy · doi:10.1007/s40121-021-00552-1 · Cited by 6 · see citing works
“Analysis of the Primary and Secondary Outcomes Even with a number of just ten clusters, multilevel modeling may result in unbiased estimations for the regression coefficients and standard errors [47].”
Background
Retention of internal teat sealants over the dry period and their efficacy in reducing clinical and subclinical mastitis at calving
Bates, A., King, C., Dhar, M., Fitzpatrick, C., & Laven, R. A. (2022) · Journal of Dairy Science · doi:10.3168/jds.2021-21585 · Cited by 6 · see citing works
A mixed model approach to estimate the survivor average causal effect in cluster‐randomized trials
Wang, W., Tong, G., Hirani, S. P., Newman, S. P., Halpern, S. D., Small, D. S., Li, F., & Harhay, M. O. (2023) · Statistics in Medicine · doi:10.1002/sim.9939 · Cited by 6 · see citing works
“40,41 We randomly sample the treated clusters and the control clusters separately.”
Method
Multisite educational trials: estimating the effect size and its confidence intervals
Singh, A., Uwimpuhwe, G., Li, M., Einbeck, J., Higgins, S., & Kasim, A. (2021) · International Journal of Research & Method in Education · doi:10.1080/1743727x.2021.1882416 · Cited by 6 · see citing works
“However, bootstrapping in education trials requires further research to fully understand the implication of bootstrapping at the level of school or pupils in cluster or multisite randomized trials (Gehlbach et al. 2016; Huang 2018).”
Method
Common odds ratio test and interval estimation for stratified bilateral and unilateral data
Hua, S., & Ma, C. (2024) · Statistical Methods in Medical Research · doi:10.1177/09622802241267357 · Cited by 5 · see citing works
“For example, cluster bootstrapping, 8,9 though computationally intensive, provides robust standard error and point estimates when dealing with correlated data.”
Multiply robust generalized estimating equations for cluster randomized trials with missing outcomes
Rabideau, D. J., Li, F., & Wang, R. (2024) · Statistics in Medicine · doi:10.1002/sim.10027 · Cited by 5 · see citing works
“…procedure, we resampled entire clusters separately within each randomized group (ie, resampled K ∕ 2 clusters in the treatment arm with replacement and resampled K ∕ 2 clusters in the control group with replacement) to maintain an equal number of clusters per arm across every bootstrap replicate.”
Method
Estimating infants’ language exposure: A comparison of random and volume sampling from daylong recordings collected in a bilingual community
Ramírez, N. F., & Hippe, D. S. (2024) · Infant Behavior and Development · doi:10.1016/j.infbeh.2024.101943 · Cited by 6 · see citing works
Hypertension and one-year risk of all-cause mortality among women with treated HIV in the United States
Sadinski, L. M., Westreich, D., Edmonds, A., Breger, T., Cole, S., Ramirez, C., Brown, T., Ofotokun, I., Konkle-Parker, D., Kassaye, S. G., et al. (2022) · AIDS (London) · doi:10.1097/qad.0000000000003461 · Cited by 3 · see citing works
“We calculated Wald 95% confidence intervals (95% CIs) by employing nonparametric bootstrap methods [43].”
BackgroundMethod
Target trial framework for determining the effect of changes in training load on injury risk using observational data: a methodological commentary
Wang, C., Kaufman, J. S., Steele, R. J., & Shrier, I. (2024) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2024-002037 · Cited by 6 · see citing works
“39 55 In both a repeated measures RCT and observational study, we would have to account for repeated measures in the analyses (eg, through cluster bootstrapping, 56 mixed models, 57 58 or generalised estimating equations 59 ).”
Method
The Target Study: A Conceptual Model and Framework for Measuring Disparity
Jackson, J. W., Hsu, Y., Greer, R. C., Boonyasai, R. T., & Howe, C. J. (2025) · Sociological Methods & Research · doi:10.1177/00491241251314037 · Cited by 12 · see citing works
Cluster-robust standard errors with three-level data
Huang, F. L., & Zhang, B. (2025) · Communication in Statistics- Theory and Methods · doi:10.1080/03610926.2025.2461609 · Cited by 6 · see citing works
“However, methodologists (Huang 2016; McNeish, Stapleton, and Silverman 2017) have pointed out that several alternative approaches for the analysis of nested data are possible such as the use of multilevel models (Raudenbush and Bryk 2002), generalized estimating equations (Huang 2022), or cluster…”
Self-citation
Evaluating Effects of Multilevel Interventions on Disparity in Health and Healthcare Decisions
Jackson, J. W., Hsu, Y., Zalla, L. C., Carson, K. A., Marsteller, J. A., Cooper, L. A., & the RICH LIFE Project Investigators (2024) · Prevention Science · doi:10.1007/s11121-024-01677-8 · Cited by 5 · see citing works
“For standard errors and confidence intervals that account for the hierarchical structure of the data, we suggest a non-parametric, balanced, stratified cluster bootstrap procedure (Davison & Hinkley, 1997; Field & Welsh, 2007; Gleason, 1988; Huang, 2018; Ren et al., 2010).”
Polyomavirus Antibodies for Merkel Cell Carcinoma Recurrence Detection
Gunnell, L., Hippe, D. S., Park, S. Y., et al. (2025) · JAMA Dermatology · doi:10.1001/jamadermatol.2025.3155 · Cited by 3 · see citing works
Political trust and informal traders in African cities
Resnick, D., & Sivasubramanian, B. (2023) · The Journal of Modern African Studies · doi:10.1017/s0022278x23000125 · Cited by 3 · see citing works
Diffusion weighted imaging for improving the diagnostic performance of screening breast MRI: impact of apparent diffusion coefficient quantitation methods and cutoffs
Biswas, D., Hippe, D. S., Winter, A. M., et al. (2024) · Frontiers in Oncology · doi:10.3389/fonc.2024.1437506 · Cited by 3 · see citing works
The Role of Opportunity to Learn and School Socioeconomic Composition in Reducing Racial and Gendered Disparities in Mathematics Achievement
Womack, T. A., Palardy, G. J., & Park, S. (2024) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2024.2342850 · Cited by 2 · see citing works
“The procedure and its rationale are described in Huang (2018).”
Multicenter evaluation of machine and deep learning methods to predict glaucoma surgical outcomes
Barry, S., & Wang, S. Y. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1636410 · Cited by 2 · see citing works
“Confidence intervals were calculated via clustered bootstrapping (Ying et al., 2022; Huang, 2018) to account for within-patient clustering, as some patients underwent multiple surgeries and thus contributed multiple observations to the dataset.”
The effect of long-term adherence to physical activity recommendations in midlife on plasma proteins associated with frailty in the Atherosclerosis Risk in Communities (ARIC) study
Liu, F., Schrack, J. A., Walker, K. A., Walston, J. D., Mathias, R. A., Griswold, M. E., Palta, P., Windham, B. G., & Jackson, J. W. (2025) · European Journal of Epidemiology · doi:10.1007/s10654-025-01282-1 · Cited by 2 · see citing works
“We estimated the 95% confidence intervals using 1000 samples from the non-parametric cluster bootstrap [45], where individuals were sampled with replacement and all of their follow-up observations were retained.”
Time to Enhancement Measured From Ultrafast Dynamic Contrast-Enhanced MRI for Improved Breast Lesion Diagnosis
Chen, Y. A., Kazerouni, A. S., Phelps, M. D., et al. (2025) · Journal of Breast Imaging · doi:10.1093/jbi/wbae089 · Cited by 2 · see citing works
Challenges in estimating effects of hypothetical interventions on resources patterned by structural racism: an example in a rural North Carolina Medicaid population
Dissanayake, M. V., Jackson, J. W., Martin, C. L., Urrutia, R. P., Funk, M. J., & Wood, M. E. (2025) · American Journal of Epidemiology · doi:10.1093/aje/kwaf072 · Cited by 2 · see citing works
Inference of random effects for linear mixed-effects models with a fixed number of clusters
Chang, C., Huang, H., & Ing, C. (2022) · Annals of the Institute of Statistical Mathematics · doi:10.1007/s10463-022-00825-7 · Cited by 2 · see citing works
“The research of Hsin-Cheng Huang is supported by ROC Ministry of Science and Technology grant MOST 106-2118-M-001-002-MY3.”
BackgroundInfluential
Best Linear Unbiased Prediction of Latent Means in Three-Level Data
Aydın, B., & Algina, J. J. (2021) · The Journal of Experimental Education · doi:10.1080/00220973.2021.1873088 · Cited by 2 · see citing works
“This clustered structure is often accounted by model based or design based methods, with multilevel modeling (also called hierarchical linear modeling and mixed effects models), a model based method, argued to be the predominant choice in the field of social sciences (Gardiner et al., 2009; Hoffman & Rovine, 2007; Hua…”
Method
Sharing work and food within the household: Intra-couple time allocation effects on nutritional outcomes in rural Telangana, India
Aderanti, O., Srinivasan, C. S., & Zanello, G. (2025) · Feminist Economics · doi:10.1080/13545701.2025.2548289 · Cited by 1 · see citing works
“Hitomi Komatsu, Hazel J. Malapit, and Sophie Theis (2018) found that women’s agricultural time use is associated with a reduction in the consumption of diverse diets among women in Mozambique.”
Blood pressure trajectories among people with and without HIV in a U.S.-based prospective cohort study
Sadinski, L. M., Edwards, J. K., Herce, M. E., et al. (2025) · AIDS · doi:10.1097/qad.0000000000004387 · Cited by 1 · see citing works
Evaluating ESCO skill classifiers
Marconi, G., Baer, I., Silveira, M. D., et al. (2025) · Statistical Journal of the IAOS · doi:10.1177/18747655251395229 · Cited by 2 · see citing works
“35,36 In our case, we need to consider clustering of the standard error at the skill and at the job ad level.”
Outcomes of patients with hypertensive emergency managed in intensive vs intermediate care settings: a multihospital retrospective cohort study
Hochberg, C. H., Yan, L., Card, M. E., Berry, S. A., Brodie, D., Rotello, L. C., Nader, A., Chatterjee, S., Sahetya, S. K., Iwashyna, T. J., et al. (2026) · Annals of the American Thoracic Society · doi:10.1093/annalsats/aaoaf048 · Cited by 1 · see citing works
CD8 PET imaging with [ 89 Zr]Zr-berdoxam-crefmirlimab in patients with solid tumors: the Phase II iCorrelate study
Postow, M. A., Zang, P. D., Pal, S. K., Mauguen, A., Farwell, M. D., Gordon, M. S., Hays, D., Wong, J. Y. C., Chen, D. L., Ulaner, G. A., et al. (2026) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2025-012208 · Cited by 2 · see citing works
Longitudinal short-term assessments of single- and multi-protein disease activity biomarkers to determine reliability and the minimum detectable change in people with clinically stable RRMS
McCurdy, S., Sheng, E., Jalaleddini, K., et al. (2026) · Multiple Sclerosis Journal · doi:10.1177/13524585261439680 · Cited by 1 · see citing works
“95% Confidence intervals for these estimates were obtained via cluster bootstrapping, 18 using the 2.5 and 97.5 quantiles from 2,000 replicates.”
Influential
Assessing the dynamic impacts of tree, stand, and climatic factors on individual tree mortality using survival analysis
Senevirathne, D. M., Yang, S.-I., Zhao, D., et al. (2026) · Forest Ecology and Management · doi:10.1016/j.foreco.2026.123935 · Cited by 1 · see citing works
Nonselective Intra-arterial CT Angiography Versus Selective Digital Subtraction Angiography for Detecting Neovascularization During Transarterial Embolization for Adhesive Capsulitis
Liao, W.-C., Huang, H.-Y., Chen, Y.-H., Huang, H.-B., Liaw, W., & Hsu, Y. (2026) · CardioVascular and Interventional Radiology · doi:10.1007/s00270-026-04533-0 · Cited by 1 · see citing works
“Between-modality differences with 95% CIs were obtained via patient-level cluster bootstrap resampling (B = 1,000) [18], resampling patients and retaining all sites to account for within-patient correlation.”
Examining the Relationship Between Multilevel Resilience Resources and Cardiovascular Disease Incidence, Overall and by Psychosocial Risks, Among Participants in the Jackson Heart Study, the Multi-Ethnic Study of Atherosclerosis, and the Mediators of Atherosclerosis in South Asians Living in America (MASALA) Study
Park, J. W., Dulin, A., Scarpaci, M. M., Dionne, L. A., Needham, B. L., Sims, M., Kanaya, A. M., Kandula, N. R., Loucks, E. B., Fava, J. L., et al. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad159 · Cited by 1 · see citing works
“To account for within-neighborhood clustering of CVD events in the secondary analyses, we used cluster boot-strapping with 200 repetitions to obtain the 95% confidence intervals (CIs) (40, 43, 44).”
Method
Identification of Kappaphycus alvarezii seaweed based on phylogenetic and carrageenan conten
Satriani, G. I., Soelistyowati, D. T., Alimuddin, A., et al. (2024) · Jurnal Akuakultur Indonesia · doi:10.19027/jai.23.1.1-11 · Cited by 1 · see citing works
“The method neighbor-joining (NJ) and replication bootstrap 1000 times constructs the phylogenetic tree (Huang, 2018).”
Method
Students’ mastery goal orientation, academic achievement and education-to-employment transition attitudes. Evidence from 30 countries
Cherkezova, S. (2024) · Environment and Social Psychology · doi:10.54517/esp.v9i5.2313 · Cited by 3 · see citing works
Design of field trials for the evaluation of transmissible vaccines in animal populations
Sheen, J. K., Kennedy‐Shaffer, L., Levy, M. Z., & Metcalf, C. J. E. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012779 · Cited by 1 · see citing works
“Alternatively, investigators may also include all originally sampled animals from each resampled cluster, among other resampling approaches [46].”
Learning Gaussian graphical models from correlated data
Song, Z., Gunn, S., Monti, S., et al. (2025) · Frontiers in Systems Biology · doi:10.3389/fsysb.2025.1589079 · Cited by 1 · see citing works
“This is consistent with the well-known limitation that the bootstrap estimate of the standard error is not accurate when the number of clusters is small and the test statistics tends to be too liberal (Huang, 2018).”
Men’s engagement with mental health treatment: towards inclusive interventions in low-income contexts
Lakin, D. P. (2019)
Upstrap for Estimating Power and Sample Size in Complex ModelsPreprint
Karas, M., & Crainiceanu, C. M. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.08.21.457220
A novel application of multilevel SEM: Teaching quality as mediator between intervention and student achievement
Tian, R., Stojanovski, E., & Miller, D. (2023) · MODSIM · doi:10.36334/modsim.2023.tian126
Capturing Session-to-Session Dynamics of Learning and Forgetting: Testing the Limits of Knowledge Tracing ModelsPreprint
Schuetze, B. A., Yan, V. X., & Carvalho, P. F. (2024) · doi:10.31234/osf.io/yu8tr
Capturing Session-to-Session Dynamics of Learning and Forgetting: Testing the Limits of Knowledge Tracing ModelsPreprint
Schuetze, B. A., Yan, V. X., & Carvalho, P. F. (2025) · doi:10.31234/osf.io/yu8tr_v2
Capturing Session-to-Session Dynamics of Learning and Forgetting: Testing the Limits of Knowledge Tracing Models
Schuetze, B. A., Yan, V. X., & Carvalho, P. F. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00508-3
“We used cluster bootstrapping (Huang, 2018) with 1000 replicates to calculate mean RMSE, AUC, classification accuracy, and bias to compute appropriate 95% confidence intervals of these statistics.”
Using Previous Longitudinal Group Randomized Rural Weight Loss Study Data to Design A Prospective Rural Weight Loss Trial
Brown, A. R., Gajewski, B. J., Mayo, M. S., Ellerbeck, E. F., & Befort, C. A. (2025) · International Journal of Clinical Biostatistics and Biometrics · doi:10.23937/2469-5831/1510058
Multiparametric breast MRI to problem-solve mammographically detected suspicious calcifications
Park, V. Y., Hippe, D. S., Kazerouni, A. S., et al. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.112467
Availability of Onsite Childcare Is Associated With Female Resident Representation: A Cross-Sectional Study
Nikolla, D. A., Bowers, K. M., Mudrakola, V., et al. (2025) · Women s Health Issues · doi:10.1016/j.whi.2025.10.006
Unrevealing the sequence of dysphagia progression in ALS: an event-based, FEES-driven staging approach
Milella, G., Fiorella, M. L., Velucci, V., et al. (2025) · Journal of Neural Transmission · doi:10.1007/s00702-025-03069-w
De-escalation techniques on daily duty: a body-worn camera analysis of police–civilian interactions in the Netherlands
van Lith, L., Ejbye-Ernst, P., Borghuis, E. E., & Hutter, R. I. (2026) · Policing: An International Journal · doi:10.1108/pijpsm-08-2025-0146
Pragmatic measurement of mechanisms: Does use of coping strategies mediate the effects of a teacher stress intervention?
Eddy, C. L., Herman, K. C., Huang, F. L., Weigand, T., Venkat, S., & Conway, C. (2026) · Journal of School Psychology · doi:10.1016/j.jsp.2026.101561
Self-citation
Machine learning–based skin nerve morphometry for diabetic neuropathy: diagnostic and clinical implications
Hsueh, H., Wu, Y., Chuang, T., et al. (2026) · Brain Communications · doi:10.1093/braincomms/fcag113
“33,34 Significance was defined when the confidence interval of the difference did not cross zero, with P values being adjusted for multiple comparisons by using the Holm method.”
Applying the Target Study Conceptual Model to Measure Racial and Ethnic Disparities in Hypertension Treatment Intensification
Meche, A., Boonyasai, R. T., Hsu, Y., Greer, R. C., Mehta, H., Alexander, G. C., Segal, J. B., Cooper, L. A., & Jackson, J. W. (2026) · Epidemiology · doi:10.1097/ede.0000000000001964
Crown-based biomass indicators and dual topographic regulation of carbon allocation in alpine Rhododendron shrublands on the Tibetan PlateauPreprint
Changsheng, L., Daqing, L., Jiangping, F., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6506531
Different Behavioral Measures of Conditioned Magazine Activity Can Tell Different Stories about Brain Function
Volz, S. G., Loewinger, G., Márquez, I., Fevola, S., Kang, M., Reverte, I., Krishnan, A., Gardner, M., Iordanova, M. D., & Esber, G. R. (2026) · eNeuro · doi:10.1523/eneuro.0560-24.2026
“One possible approach would be to construct con ﬁ dence intervals using a cluster-bootstrapping procedure (Huang, 2018, and references therein) that resamples at both the subject and trial levels, re-estimates subject-speci ﬁ c PCs, and ﬁ ts the model to each bootstrap sample.”
ctDNA or Merkel Virus Antibodies for Surveillance of Merkel Cell Carcinoma Recurrence
Thakuria, M., Akaike, T., Silk, A. W., et al. (2026) · JAMA Dermatology · doi:10.1001/jamadermatol.2026.2153
Crown-based biomass indicators and dual topographic regulation of biomass allocation in alpine Rhododendron shrublands on the Tibetan plateau
Changsheng, L., Luo, D., Jiangping, F., et al. (2026) · Ecological Indicators · doi:10.1016/j.ecolind.2026.115139
Bridging the Gap? Liveübertragungen in Kreistagen und Stadträten als Datengrundlage für die politikwissenschaftliche Forschung
Reimers, P. (2026) · Politische Vierteljahresschrift · doi:10.1007/s11615-026-00672-8
Are Humans and LLMs Confused by the Same Code? An Empirical Study on Fixation-Related Potentials and LLM Perplexity
Abdelsalam, Y., Peitek, N., Maurer, A.-M., Toneva, M., & Apel, S. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773219 · Cited by 3 · see citing works
“The procedure consists of four main steps.”

Authors: Huang, F. L.
Affiliations: University of Missouri (US)
Corresponding author: Huang, Francis L.
ORCID: Huang, Francis L.
Venue: Educational and Psychological Measurement
Publisher: SAGE Publications
Published: 2016-11-24
Volume: 78
Issue: 2
Pages: 297-318
ISSN: 0013-1644 (ISSN-L)
1552-3888
DOI: 10.1177/0013164416678980
PMC: PMC5965657
OpenAlex: W2554437640
Cited by: 107
References: 65
Citations by year: 2026: 22 · 2025: 22 · 2024: 18 · 2023: 12 · 2022: 10 · 2021: 13 · 2020: 3 · 2019: 2 · 2018: 2 · 2017: 3
FWCI: 2.36
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Educational and Psychological Measurement (publishedVersion)
Europe PMC (PubMed Central) (publishedVersion)
PubMed Central (publishedVersion)
PubMed (publishedVersion)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords: small number of clusters, nested data, multilevel models, standard errors, intraclass correlation coefficient, cluster size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Using cluster bootstrapping to analyze nested data with a few clusters

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks