The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
How Methodological Features Affect Effect Sizes in Education
Abstract
As evidence becomes increasingly important in educational policy, it is essential to understand how research design might contribute to reported effect sizes in experiments evaluating educational programs. A total of 645 studies from 12 recent reviews of evaluations of preschool, reading, mathematics, and science programs were studied. Effect sizes were roughly twice as large for published articles, small-scale trials, and experimenter-made measures, compared to unpublished documents, large-scale studies, and independent measures, respectively. Effect sizes were significantly higher in quasi-experiments than in randomized experiments. Excluding tutoring studies, there were no significant differences in effect sizes between elementary and middle/high studies. Regression analyses found that effects of all factors maintained after controlling for all other factors. Explanations for the effects of methodological features on effect sizes are discussed, as are implications for evidence-based policy.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In addition, published studies in higher rated journals may report larger effects than unpublished reports due to biases in publishing only the stronger associations and significant results (Cheung & Slavin, 2016).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Cheung & Slavin, 2016
In the citing paper
“major factors that have consistently shown to affect meta‐analytic results in other fields of research are study characteristics, such as the year and quality of the study (Cheung & Slavin, 2016).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Cheung & Slavin, 2016
In the citing paper
“pre-to post effects generally are much larger than in randomized controlled studies [33,34]”
From Virtual reality exposure therapy for fear of spiders: an open trial and feasibility study of a new treatment for arachnophobia (2024) · cites this paper as [33]
In the citing paper
“Similar effect sizes are substantial; Cheung and Slavin (2016) find, in their meta-analysis of 197 RCTs, an average effect size on academic achievement of 0.16 standard deviations.”
From The effect of school closures on standardised student test outcomes (2022) · cites this paper as Cheung and Slavin (2016)
Citations
Cites
(2000) · Cited by 9,198
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,896
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033
Is working memory training effective? A meta-analytic review
Melby-Lervåg, M., & Hulme, C. (2013) · Developmental Psychology · doi:10.1037/a0028228 · Cited by 1,782
Espinosa, J. C. (2004) · Cited by 939
Cheung, A. C. K., & Slavin, R. E. (2013) · Educational Research Review · doi:10.1016/j.edurev.2013.01.001 · Cited by 792
Self-citationFerguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
Edmonds, M. S., Vaughn, S., Wexler, J., Reutebuch, C., Cable, A., Tackett, K. K., & Schnakenberg, J. W. (2009) · Review of Educational Research · doi:10.3102/0034654308325998 · Cited by 599
Grey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub3 · Cited by 598
Slavin, R. E. (2008) · Educational Researcher · doi:10.3102/0013189x08314117 · Cited by 560
Self-citation
Research works printed but not linked: 7
- Baron J., 2003, How to assess whether an educational intervention has been “proven effective” in rigorous research
- Buck S., 2015, Why government needs more randomized control trials: Refuting the myths
- Liao Y. K. C., 1999, Journal of Educational Multimedia and Hypermedia, 8, 255
- Nussle J., 2014, Moneyball for government
- Pearson P. D., 2005, The effects of technology on reading performance in the middle-school grades: A meta-analysis with recommendation for policy
- Slavin R. E., 2013, Education Week, 32, 24
- What Works Clearinghouse, 2014, Procedures and standards handbook
Cited by
Sorted by citations
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
Mindsets: A View From Two Eras
Dweck, C. S., & Yeager, D. S. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691618804166 · Cited by 1,259
What can be learned from growth mindset controversies?
Yeager, D. S., & Dweck, C. S. (2020) · American Psychologist · doi:10.1037/amp0000794 · Cited by 868
Cipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
Effects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis
van Alten, D. C. D., Phielix, C., Janssen, J. J. H. M., & Kester, L. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.05.003 · Cited by 735
Metadata
- Authors
- Cheung, A. C. K., & Slavin, R. E.
- Affiliations
- Chinese University of Hong Kong (HK)
- Johns Hopkins University (US)
- Venue
- Educational Researcher
- Published
- 2016-06
- Volume
- 45
- Issue
- 5
- Pages
- 283-292
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2470242385
- Cited by
- 529
- References
- 37
- Citations by year
- 2026: 61 · 2025: 57 · 2024: 57 · 2023: 56 · 2022: 62 · 2021: 57 · 2020: 94 · 2019: 36 · 2018: 23 · 2017: 18 · 2016: 8
- FWCI
- 26.47
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Researcher (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Education › Educational Research and Methods
- Keywords
- effect size, research design, methodological characteristics, educational program evaluation, randomized experiments, quasi-experiments, unpublished research, preschool programs, reading programs, mathematics programs, science programs, evidence-based policy, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
