The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
How Methodological Features Affect Effect Sizes in Education
Abstract
As evidence becomes increasingly important in educational policy, it is essential to understand how research design might contribute to reported effect sizes in experiments evaluating educational programs. A total of 645 studies from 12 recent reviews of evaluations of preschool, reading, mathematics, and science programs were studied. Effect sizes were roughly twice as large for published articles, small-scale trials, and experimenter-made measures, compared to unpublished documents, large-scale studies, and independent measures, respectively. Effect sizes were significantly higher in quasi-experiments than in randomized experiments. Excluding tutoring studies, there were no significant differences in effect sizes between elementary and middle/high studies. Regression analyses found that effects of all factors maintained after controlling for all other factors. Explanations for the effects of methodological features on effect sizes are discussed, as are implications for evidence-based policy.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In addition, published studies in higher rated journals may report larger effects than unpublished reports due to biases in publishing only the stronger associations and significant results (Cheung & Slavin, 2016).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Cheung & Slavin, 2016
In the citing paper
“major factors that have consistently shown to affect meta‐analytic results in other fields of research are study characteristics, such as the year and quality of the study (Cheung & Slavin, 2016).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Cheung & Slavin, 2016
In the citing paper
“pre-to post effects generally are much larger than in randomized controlled studies [33,34]”
From Virtual reality exposure therapy for fear of spiders: an open trial and feasibility study of a new treatment for arachnophobia (2024) · cites this paper as [33]
In the citing paper
“Similar effect sizes are substantial; Cheung and Slavin (2016) find, in their meta-analysis of 197 RCTs, an average effect size on academic achievement of 0.16 standard deviations.”
From The effect of school closures on standardised student test outcomes (2022) · cites this paper as Cheung and Slavin (2016)
Citations
Cites
(2000) · Cited by 9,198
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,896
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033
Is working memory training effective? A meta-analytic review
Melby-Lervåg, M., & Hulme, C. (2013) · Developmental Psychology · doi:10.1037/a0028228 · Cited by 1,782
Espinosa, J. C. (2004) · Cited by 939
Cheung, A. C. K., & Slavin, R. E. (2013) · Educational Research Review · doi:10.1016/j.edurev.2013.01.001 · Cited by 792
Self-citationFerguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603
Edmonds, M. S., Vaughn, S., Wexler, J., Reutebuch, C., Cable, A., Tackett, K. K., & Schnakenberg, J. W. (2009) · Review of Educational Research · doi:10.3102/0034654308325998 · Cited by 599
Grey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub3 · Cited by 598
Slavin, R. E. (2008) · Educational Researcher · doi:10.3102/0013189x08314117 · Cited by 560
Self-citationA Meta-analysis of the Effects of Computer Technology on School Students’ Mathematics Learning
Li, Q., & Ma, X. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9125-8 · Cited by 558
Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W. (2008) · Journal of Research on Educational Effectiveness · doi:10.1080/19345740802400072 · Cited by 477
Effective Programs in Elementary Mathematics: A Best-Evidence Synthesis
Slavin, R. E., & Lake, C. (2008) · Review of Educational Research · doi:10.3102/0034654308317473 · Cited by 407
Self-citationThe Relationship Between Sample Sizes and Effect Sizes in Systematic Reviews in Education
Slavin, R. E., & Smith, D. (2009) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373709352369 · Cited by 390
Self-citationEffective programs for struggling readers: A best-evidence synthesis
Slavin, R. E., Lake, C., Davis, S., & Madden, N. A. (2011) · Educational Research Review · doi:10.1016/j.edurev.2010.07.002 · Cited by 353
Self-citationEffective Reading Programs for the Elementary Grades: A Best-Evidence Synthesis
Slavin, R. E., Lake, C., Chambers, B., Cheung, A., & Davis, S. (2009) · Review of Educational Research · doi:10.3102/0034654309341374 · Cited by 352
Self-citationHow features of educational technology applications affect student reading outcomes: A meta-analysis
Cheung, A. C. K., & Slavin, R. E. (2012) · Educational Research Review · doi:10.1016/j.edurev.2012.05.002 · Cited by 307
Self-citationEffective Reading Programs for Middle and High Schools: A Best‐Evidence Synthesis
Slavin, R. E., Cheung, A., Groff, C., & Lake, C. (2008) · Reading Research Quarterly · doi:10.1598/rrq.43.3.4 · Cited by 307
Self-citationEffective Programs in Middle and High School Mathematics: A Best-Evidence Synthesis
Slavin, R. E., Lake, C., & Groff, C. (2009) · Review of Educational Research · doi:10.3102/0034654308330968 · Cited by 300
Self-citationShould Unpublished Data Be Included in Meta-analyses?: Current Convictions and Controversies
Cook, D. J. (1993) · JAMA · doi:10.1001/jama.1993.03500210049030 · Cited by 259 · cited 5×
“The findings are consistent with previous studies in social science, psychological science, and medicine (Cook et al., 1993; Ferguson & Heene, 2012; Glass et al., 1981; Hopewell, McDonald, Clarke, & Egger, 2007; Lipsey & Wilson, 1993). For example, Lipsey and Wilson (1993) found that across 92 metaanalyses of psycholo…”
BackgroundResultInfluentialDavidian, M., & Giltinan, D. M. (2017) · Routledge eBooks · doi:10.1201/9780203745502-3 · Cited by 221
Interventions for Adolescent Struggling Readers: A Meta-Analysis with Implications for Practice.
Scammacca, N. K., Roberts, G., Vaughn, S. R., Edmonds, M. S., Wexler, J., Reutebuch, C. K., & Torgesen, J. K. (2007) · Cited by 206
Cheung, A., & Slavin, R. E. (2013) · Reading Research Quarterly · doi:10.1002/rrq.50 · Cited by 175
Self-citationShow Me the Evidence: Obama's Fight for Rigor and Results in Social Policy
Haskins, R., & Margolis, G. (2014) · Project Muse (Johns Hopkins University) · Cited by 170
Toward Reform of Program Evaluation: Aims, Methods, and Institutional Arrangements.
Cain, P. S., & Cronbach, L. J. (1982) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2068404 · Cited by 167
Cheung, A., & Slavin, R. E. (2012) · Review of Educational Research · doi:10.3102/0034654312465472 · Cited by 162
Self-citationEffects of the Attributes of Educational Interventions on Students’ Academic Performance
de Boer, H., Donker, A. S., & van der Werf, G. P. C. (2014) · Review of Educational Research · doi:10.3102/0034654314540006 · Cited by 161
Methods of Instructional Improvement in Algebra
Rakes, C. R., Valentine, J. C., McGatha, M. B., & Ronau, R. N. (2010) · Review of Educational Research · doi:10.3102/0034654310374880 · Cited by 156
Toward reform of program evaluation: Aims, methods, and institutional arrangements
Weiss, C. H. (1981) · Evaluation and Program Planning · doi:10.1016/0149-7189(81)90038-0 · Cited by 152
Experimental evaluations of elementary science programs: A best-evidence synthesis
Slavin, R. E., Lake, C., Hanley, P., & Thurston, A. (2014) · Journal of Research in Science Teaching · doi:10.1002/tea.21139 · Cited by 137
Self-citationHeinsman, D. T., & Shadish, W. R. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.154 · Cited by 125
Measures Inherent to Treatments in Program Effectiveness Reviews
Slavin, R., & Madden, N. A. (2011) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.558986 · Cited by 124 · cited 2×
“As another example of the correlation between methodology and outcomes, Slavin and Madden (2011) examined effect sizes of reading and math studies that met the standards of the WWC with or without reservations.”
BackgroundMethodSelf-citationEffects of hypermedia on students' achievement
Liao, Y.-K. C. (1999) · Journal of educational multimedia and hypermedia · doi:10.5555/339278.339279 · Cited by 98
Chambers, B., Cheung, A., & Slavin, R. E. (2016) · Educational Research Review · doi:10.1016/j.edurev.2016.03.003 · Cited by 96
Self-citationTorgerson, C. J. (2006) · Journal of Research in Reading · doi:10.1111/j.1467-9817.2006.00318.x · Cited by 59
Critical Issues and Common Pitfalls in Designing and Conducting Impact Studies in Education
Song, M., & Herman, R. (2010) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373710373389 · Cited by 56
Research works printed but not linked: 7
- Baron J., 2003, How to assess whether an educational intervention has been “proven effective” in rigorous research
- Buck S., 2015, Why government needs more randomized control trials: Refuting the myths
- Liao Y. K. C., 1999, Journal of Educational Multimedia and Hypermedia, 8, 255
- Nussle J., 2014, Moneyball for government
- Pearson P. D., 2005, The effects of technology on reading performance in the middle-school grades: A meta-analysis with recommendation for policy
- Slavin R. E., 2013, Education Week, 32, 24
- What Works Clearinghouse, 2014, Procedures and standards handbook
Cited by
Sorted by citations
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
Mindsets: A View From Two Eras
Dweck, C. S., & Yeager, D. S. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691618804166 · Cited by 1,259
What can be learned from growth mindset controversies?
Yeager, D. S., & Dweck, C. S. (2020) · American Psychologist · doi:10.1037/amp0000794 · Cited by 868
Cipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
Effects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis
van Alten, D. C. D., Phielix, C., Janssen, J. J. H. M., & Kester, L. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.05.003 · Cited by 735
Metadata
- Authors
- Cheung, A. C. K., & Slavin, R. E.
- Affiliations
- Chinese University of Hong Kong (HK)
- Johns Hopkins University (US)
- Venue
- Educational Researcher
- Published
- 2016-06
- Volume
- 45
- Issue
- 5
- Pages
- 283-292
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2470242385
- Cited by
- 529
- References
- 37
- Citations by year
- 2026: 61 · 2025: 57 · 2024: 57 · 2023: 56 · 2022: 62 · 2021: 57 · 2020: 94 · 2019: 36 · 2018: 23 · 2017: 18 · 2016: 8
- FWCI
- 26.47
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Researcher (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Education › Educational Research and Methods
- Keywords
- effect size, research design, methodological characteristics, educational program evaluation, randomized experiments, quasi-experiments, unpublished research, preschool programs, reading programs, mathematics programs, science programs, evidence-based policy, publication bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
