Statistical power and optimal design for multisite randomized trials.
Abstract
The multisite trial, widely used in mental health research and education, enables experimenters to assess the average impact of a treatment across sites, the variance of treatment impact across sites, and the moderating effect of site characteristics on treatment efficacy. Key design decisions include the sample size per site and the number of sites. To consider power implications, this article proposes a standardized hierarchical linear model and uses rules of thumb similar to those proposed by J. Cohen (1988) for small, medium, and large effect sizes and for small, medium, and large treatment-by-site variance. Optimal allocation of resources within and between sites as a function of variance components and costs at each level are also considered. The approach generalizes to quasiexperiments with a similar structure. These ideas are illustrated with newly developed software.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A power analysis indicated that 67 total participants were needed for 80% power to detect a between-group effect size of .70 at 12-month follow-up, and 126 were needed for an effect size of .50.
“Power analyses, conducted in Optimal Design (see Raudenbush & Liu, 2000), indicated that to reach 80% power, a cross sectional between-group difference (e.g., at 12mFU) with an effect size of .70 required 67 total Ps, whereas a between-group effect size of .50 required 126 total Ps.”
From Randomized clinical trial of cognitive behavioral therapy (CBT) versus acceptance and commitment therapy (ACT) for mixed anxiety disorders (2012) · cites this paper as Raudenbush & Liu, 2000
Cites
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · see citing works
Using SAS PROC MIXED to Fit Multilevel Models, Hierarchical Models, and Individual Growth Models
Singer, J. D. (1998) · Journal of Educational and Behavioral Statistics · doi:10.2307/1165280 · Cited by 1,092 · see citing works
MIXOR: a computer program for mixed-effects ordinal regression analysis
Hedeker, D., & Gibbons, R. D. (1996) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(96)01720-8 · Cited by 408 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-013-0528-y · Cited by 12,038 · see citing works
“(Bloom and Michalopoulos 2013; Raudenbush and Liu 2000).”
Background<scp>SIMR</scp> : an R package for power analysis of generalized linear mixed models by simulation
Green, P., & MacLeod, C. J. (2016) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12504 · Cited by 2,467 · see citing works
Kirkman, B. L., Chen, G., Farh, J.-L., Chen, Z. X., & Lowe, K. B. (2009) · Academy of Management Journal · doi:10.5465/amj.2009.43669971 · Cited by 1,094 · see citing works
Possible selves and academic outcomes: How and when possible selves impel action.
Oyserman, D., Bybee, D., & Terry, K. (2006) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.91.1.188 · Cited by 852 · see citing works
Nezlek, J. B. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/0146167201277001 · Cited by 676 · see citing works
Design and analysis of group-randomized trials: a review of recent methodological developments
Murray, D. M., Varnell, S. P., & Blitstein, J. L. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.423 · Cited by 677 · see citing works
An Overview of Research and Evaluation Designs for Dissemination and Implementation
Brown, C. H., Curran, G., Palinkas, L. A., Aarons, G. A., Wells, K. B., Jones, L., Collins, L. M., Duan, N., Mittman, B. S., Wallace, A., et al. (2017) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031816-044215 · Cited by 509 · see citing works
Methods for sample size determination in cluster randomized trials
Rutterford, C., Copas, A., & Eldridge, S. (2015) · International Journal of Epidemiology · doi:10.1093/ije/dyv113 · Cited by 429 · see citing works
Raudenbush, S. W., & Liu, X.-F. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.387 · Cited by 455 · see citing works
Self-citationArch, J. J., Eifert, G. H., Davies, C., Vilardaga, J. C. P., Rose, R. D., & Craske, M. G. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0028310 · Cited by 443 · see citing works
A systematic review of empirical studies examining mechanisms of implementation in health
Lewis, C. C., Boyd, M. R., Walsh-Bailey, C., Lyon, A. R., Beidas, R., Mittman, B., Aarons, G. A., Weiner, B. J., & Chambers, D. A. (2020) · Implementation Science · doi:10.1186/s13012-020-00983-3 · Cited by 363 · see citing works
Pittig, A., Arch, J. J., Lam, C. W. R., & Craske, M. G. (2013) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2012.10.012 · Cited by 255 · see citing works
Yang, J., Mossholder, K. W., & Peng, T. K. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.3.681 · Cited by 196 · see citing works
Brown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., Poduska, J., Ialongo, N., Wyman, P. A., Chamberlain, P., et al. (2008) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2007.11.013 · Cited by 177 · see citing works
Arch, J. J., Ayers, C. R., Baker, A., Almklov, E., Dean, D. J., & Craske, M. G. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.01.003 · Cited by 165 · see citing works
Dziak, J. J., Nahum-Shani, I., & Collins, L. M. (2012) · Psychological Methods · doi:10.1037/a0026972 · Cited by 142 · see citing works
Therapist predictors of early patient-rated working alliance: a multilevel approach
Nissen-Lie, H. A., Monsen, J. T., & Rønnestad, M. H. (2010) · Psychotherapy Research · doi:10.1080/10503307.2010.497633 · Cited by 144 · see citing works
Cheng, C., & Cheung, M. W. L. (2005) · Journal of Personality · doi:10.1111/j.1467-6494.2004.00310.x · Cited by 130 · see citing works
Design and Analysis in Dissemination and Implementation Research
Landsverk, J., Brown, C. H., Chamberlain, P., Palinkas, L., Ogihara, M., Czaja, S., Goldhaber-Fiebert, J. D., Rolls Reutz, J. A., & Horwitz, S. M. (2012) · Dissemination and Implementation Research in HealthTranslating Science to Practice · doi:10.1093/acprof:oso/9780199751877.003.0012 · Cited by 114 · see citing works
Dynamic wait-listed designs for randomized trials: new designs for prevention of youth suicide
Brown, C. H., Wyman, P. A., Guo, J., & Peña, J. (2006) · Clinical Trials · doi:10.1191/1740774506cn152oa · Cited by 99 · see citing works
Cluster randomized trials with treatment noncompliance.
Jo, B., Asparouhov, T., Muthén, B. O., Ialongo, N. S., & Brown, C. H. (2008) · Psychological Methods · doi:10.1037/1082-989x.13.1.1 · Cited by 83 · see citing works
Po'e, E. K., Heerman, W. J., Mistry, R. S., & Barkin, S. L. (2013) · Contemporary Clinical Trials · doi:10.1016/j.cct.2013.08.013 · Cited by 78 · see citing works
Goodman, F. R., Kashdan, T. B., & İmamoğlu, A. (2021) · Emotion · doi:10.1037/emo0000750 · Cited by 51 · see citing works
Ruiz, F. J., Peña-Vargas, A., Ramírez, E. S., Suárez-Falcón, J. C., García-Martín, M. B., García-Beltrán, D. M., Henao, Á. M., Monroy-Cifuentes, A., & Sánchez, P. D. (2020) · Psychotherapy · doi:10.1037/pst0000273 · Cited by 48 · see citing works
“Because we planned to analyze the results of the trial with linear mixed models, the software Optimal Design (Raudenbush & Liu, 2000) was used to conduct the power analysis.”
MethodRobbins, L. B., Pfeiffer, K. A., Vermeesch, A., Resnicow, K., You, Z., An, L., & Wesolek, S. M. (2013) · BMC Public Health · doi:10.1186/1471-2458-13-474 · Cited by 39 · see citing works
Wiltsey Stirman, S., Finley, E. P., Shields, N., Cook, J., Haine-Schlagel, R., Burgess, J. F., Dimeff, L., Koerner, K., Suvak, M., Gutner, C. A., et al. (2017) · Implementation Science · doi:10.1186/s13012-017-0544-5 · Cited by 35 · see citing works
Goodman, F. R., Daniel, K. E., Eldesouky, L., Brown, B. A., & Kneeland, E. T. (2021) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2021.100210 · Cited by 31 · see citing works
Are people with social anxiety disorder happier alone?
Goodman, F. R., Rum, R., Silva, G., & Kashdan, T. B. (2021) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2021.102474 · Cited by 33 · see citing works
Optimal Designs for Multilevel Studies
Moerbeek, M., Breukelen, G. J. P. V., & Berger, M. P. F. (2007) · Handbook of Multilevel Analysis · doi:10.1007/978-0-387-73186-5_4 · Cited by 28 · see citing works
Forman, E. M., Berry, M. P., Butryn, M. L., Hagerman, C. J., Huang, Z., Juarascio, A. S., LaFata, E. M., Ontañón, S., Tilford, J. M., & Zhang, F. (2023) · Contemporary Clinical Trials · doi:10.1016/j.cct.2022.107029 · Cited by 24 · see citing works
Goodman, F. R., & Kashdan, T. B. (2021) · The Journal of Positive Psychology · doi:10.1080/17439760.2019.1689423 · Cited by 12 · see citing works
Hogue, A., Dauber, S., Bobek, M., Jensen-Doss, A., & Henderson, C. E. (2019) · Implementation Science · doi:10.1186/s13012-019-0874-6 · Cited by 11 · see citing works
Hogue, A., Bobek, M., MacLean, A., Porter, N., Jensen-Doss, A., & Henderson, C. E. (2019) · Trials · doi:10.1186/s13063-019-3783-8 · Cited by 9 · see citing works
Arch, J. J., Mitchell, J. L., Schmiege, S. J., Levin, M. E., Genung, S. R., Nealis, M. S., Fink, R. M., Bright, E. E., Andorsky, D. J., & Kutner, J. S. (2022) · BMC Palliative Care · doi:10.1186/s12904-022-01087-z · Cited by 8 · see citing works
“01 [70], where the effect size estimates consider the number of cohorts/intervention groups, the average sample size per group, and the intraclass correlation (ICC) assessing within-cohort similiarities.”
BackgroundJohnson, S. U., & Hoffart, A. (2019) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2361 · Cited by 7 · see citing works
- Authors
- Raudenbush, S. W., & Liu, X.
- Affiliations
- University of Michigan (US)
- ORCID
- Liu, Xiaofeng
- Venue
- Psychological Methods
- Published
- 2000
- Volume
- 5
- Issue
- 2
- Pages
- 199-213
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W1983489591
- Cited by
- 593
- References
- 3
- Citations by year
- 2026: 9 · 2025: 13 · 2024: 18 · 2023: 22 · 2022: 28 · 2021: 29 · 2020: 30 · 2019: 28 · 2018: 24 · 2017: 28 · 2016: 28 · 2015: 34 · 2014: 29 · 2013: 48 · 2012: 34
- FWCI
- 6.66
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- statistical power, sample size determination, optimal design, multilevel analysis, effect size, site effects, resource allocation, quasi-experimental design, mental health research
- MeSH terms
- Humans, Mathematical Computing, Software, Health Care Rationing, Multicenter Studies as Topic, Linear Models, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review