Statistical Power for Random Assignment Evaluations of Education Programs
The DOI matches its Crossref record; 44 of the 48 references it cites were found in a registry or an index.
Abstract
This article examines theoretical and empirical issues related to the statistical power of impact estimates for experimental evaluations of education programs. The author considers designs where random assignment is conducted at the school, classroom, or student level, and employs a unified analytic framework using statistical methods from the literature. Focusing on standardized test scores of elementary school students, this article discusses appropriate precision standards and, for each design, the required number of schools to achieve those standards using empirical values of intraclass correlations, regression R 2 values, and other parameters. Clustering effects vary by design but are typically large. Thus, large school samples are required for education trials, and many evaluations will only have sufficient power to detect precise impacts for relatively large subgroups of sites.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“estimates of 0.1 and 0.2 had been used in a review of school-based violence prevention programmes (Mytton 2006), based on the rationale for a published ICC of 0.15 for similar trials (CPPRG 1999bin Mytton 2006), and was considered a plausible yet conservative estimate for the impact of clustering at the classroom level (Schochet 2008).”
From School-based education programmes for the prevention of child sexual abuse (2015) · cites this paper as Schochet 2008
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
(1979) · Medical Entomology and Zoology · Cited by 11,252
The Design and Analysis of Clinical Experiments
Fleiss, J. L. (1999) · Wiley eBooks · doi:10.1002/9781118032923 · Cited by 4,819
The Design and Analysis of Clinical Experiments.
Hills, M., & Fleiss, J. L. (1987) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2982050 · Cited by 3,261
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 1×
“Random Assignment Evaluations impacts that have been achieved (Cohen, 1988; Lipsey & Wilson, 1993).”
Searle, S. R., Casella, G., & McCulloch, C. E. (1992) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316856 · Cited by 1,926
The Role of Premarket Factors in Black-White Wage Differences
Neal, D. A., & Johnson, W. R. (1996) · Journal of Political Economy · doi:10.1086/262045 · Cited by 1,340 · cited 1×
“…studies have indicated that a one-standard-deviation increase in either math or reading test scores for elementary school children is associated with about 8% higher earnings when the students join the labor market (Currie & Thomas, 1999; Murnane, Willet, & Levy, 1995; Neal & Johnson, 1996).”
Design and Analysis of Cluster Randomization Trials in Health Research
DONNER, A. P., & Klar, N. S. (2000) · Cited by 1,236
Ukoumunne, Gulliford, Chinn, Sterne, & Burney (1999) · Health Technology Assessment · doi:10.3310/hta3050 · Cited by 1,210
The Growing Importance of Cognitive Skills in Wage Determination
Murnane, R. J., Willett, J. B., & Levy, F. (1995) · The Review of Economics and Statistics · doi:10.2307/2109863 · Cited by 1,160
Research works printed but not linked: 11
- Bloom H, 2004, Randomizing groups to evaluate place-based programs
- Bryk A, 1992, Hierarchical linear models for social and behavioral research. Applications and data analysis methods
- Cochran W, 1963, Sampling techniques
- Gargani J, 2005, How many schools? Limits of the conventional wisdom about sample size requirements for cluster randomized trials
- Gleason P, 2004, Impact evaluation of charter school strategies. Design documents
- Hedberg E, 2004, The variance structure of academic achievement in America
- Hedges L, 2004, Correcting significance tests for clustering
- Kish L, 1965, Survey sampling
- National Center for Education Statistics, 2004, The nation’s report card: Highlights 2003
- Raudenbush S, 2004, Optimal design for longitudinal and multilevel research: Documentation for the Optimal Design software
- Schochet P, 2004, Social and Character Development (SACD) Research Program: Analysis plan for the multisite impact analysis
Cited by
Jennings, P. A., Brown, J. L., Frank, J. L., Doyle, S., Oh, Y., Davis, R., Rasheed, D., DeWeese, A., DeMauro, A. A., Cham, H., et al. (2017) · Journal of Educational Psychology · doi:10.1037/edu0000187 · Cited by 645
“…as a strategy to control between-school variability (Werthamer-Larsson, 1994), and requires fewer schools to achieve adequate statistical power to detect small to moderate effects (Blitstein, Hannan, Murray, & Shadish, 2005; Cornfield, 1978; Raudenbush, Martinez, & Spybrook, 2007; Schochet, 2008).”
BackgroundMcEwan, P. J. (2015) · Review of Educational Research · doi:10.3102/0034654314553127 · Cited by 413
“In U.S. evaluations, the recommended ICCs usually fall between Improving Learning in Primary Schools .15 and .2 for test score outcomes (Hedges & Hedberg, 2007; Schochet, 2005; What Works Clearinghouse, 2011).”
BackgroundWilson, C. D., Taylor, J. A., Kowalski, S. M., & Carlson, J. (2010) · Journal of Research in Science Teaching · doi:10.1002/tea.20329 · Cited by 339
“Pretest scores typically account for a high degree of variation in posttest scores (Schochet, 2005).”
BackgroundSchool-based education programmes for the prevention of child sexual abuse
Walsh, K., Zwi, K., Woolfenden, S., & Shlonsky, A. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004380.pub3 · Cited by 318
Yoshikawa, H., Leyva, D., Snow, C. E., Treviño, E., Barata, M. C., Weiland, C., Gomez, C. J., Moreno, L., Rolla, A., D'Sa, N., et al. (2015) · Developmental Psychology · doi:10.1037/a0038785 · Cited by 286
“This number was chosen based on an initial multilevel power analysis (Moreno & Lugo-Gil, 2007; Schochet, 2008).”
Method
Metadata
- Authors
- Schochet, P. Z.
- Affiliations
- Mathematica Policy Research (US)
- Corresponding author
- Schochet, Peter Z.
- ORCID
- Schochet, Peter Z.
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 2008-03
- Volume
- 33
- Issue
- 1
- Pages
- 62-87
- ISSN
- 1076-9986 (ISSN-L)
- 1935-1054
- OpenAlex
- W2105644528
- Semantic Scholar
- b6872bd3dff61db02d0bec71fb46991952a88100
- Cited by
- 216
- References
- 48
- Citations by year
- 2026: 7 · 2025: 3 · 2024: 9 · 2023: 10 · 2022: 11 · 2021: 10 · 2020: 13 · 2019: 12 · 2018: 11 · 2017: 11 · 2016: 28 · 2015: 18 · 2014: 13 · 2013: 15 · 2012: 13
- FWCI
- 7.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Journal of Educational and Behavioral Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Education › School Choice and Performance
- Keywords
- statistical power, random assignment, educational program evaluation, impact estimation, elementary school students, standardized test scores, cluster effect, intraclass correlation coefficient, sample size determination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistical Power for Random Assignment Evaluations of Education Programs. https://app.cochat.ai/papers/10.3102/1076998607302714
- The paper
- Schochet, P. Z. (2008). Statistical Power for Random Assignment Evaluations of Education Programs. Journal of Educational and Behavioral Statistics. https://doi.org/10.3102/1076998607302714
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.