Perspectives on Evidence-Based Research in Education—What Works? Issues in Synthesizing Educational Program Evaluations
Abstract
Syntheses of research on educational programs have taken on increasing policy importance. Procedures for performing such syntheses must therefore produce reliable, unbiased, and meaningful information on the strength of evidence behind each program. Because evaluations of any given program are few in number, syntheses of program evaluations must focus on minimizing bias in reviews of each study. This article discusses key issues in the conduct of program evaluation syntheses: requirements for research design, sample size, adjustments for pretest differences, duration, and use of unbiased outcome measures. It also discusses the need to balance factors such as research designs, effect sizes, and numbers of studies in rating the overall strength of evidence supporting each program.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RCT or QED with comparison group provide the strongest evidence that programs rather than some other factor caused the improvement in creativity or divergent thinking.
“this review only included studies adopting RCT or Quasi-Experimental Design (QED) with comparison group (i.e., random assignment and/or matching with appropriate adjustment for any pretest differences) (Campbell & Stanley, 1963), because these provide the strongest evidence that programs rather than some other factor caused the improvement in creativity or divergent thinking (Slavin, 2008;Valgeirs”
From Creativity training programs in primary education: A systematic review and meta-analysis (2022) · cites this paper as Slavin, 2008
The review included only studies based on robust evaluation research designs.
“we have only included studies based on robust evaluation research designs (Slavin, 2008;Valgeirsdottir & Onarheim, 2017)”
From Creativity training programs in primary education: A systematic review and meta-analysis (2022) · cites this paper as Slavin, 2008
The concept of online or digital learning can vary an extraordinary amount between contexts and studies.
“The concept of online or digital learning can vary an extraordinary amount between contexts and studies, and in order for readers to understand and consider whether a study is applicable to their own situation, full study design details must be provided (Bond et al., 2020; Slavin, 2008).”
From Emergency remote teaching in higher education: mapping the first global online semester (2021) · cites this paper as Slavin, 2008
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075823 · Cited by 18,768 · see citing works
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,820 · see citing works
(2002) · Cited by 13,211 · see citing works
(2000) · Cited by 9,198 · see citing works
Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs
Dehejia, R. H., & Wahba, S. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473858 · Cited by 2,395 · see citing works
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368 · see citing works
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Education and Treatment of Children
Hickman, J. A., & Wheatley, J. K. (2008) · Encyclopedia of Special Education · doi:10.1002/9780470373699.speced0741 · Cited by 1,938 · see citing works
Scientific Research in Education
National Research Council (2002) · National Academies Press eBooks · doi:10.17226/10236 · Cited by 1,750 · see citing works
The review of educational research.
(1931) · Journal of Educational Psychology · doi:10.1037/h0066448 · Cited by 1,562 · see citing works
International Reading Association (2012) · Reading Research Quarterly · doi:10.1002/(issn)1936-2722 · Cited by 1,385 · see citing works
Design and Analysis of Cluster Randomization Trials in Health Research
DONNER, A. P., & Klar, N. S. (2000) · Cited by 1,236 · see citing works
Design and Analysis of Group-Randomized Trials
Sonnad, S. S. (1999) · Medicine & Science in Sports & Exercise · doi:10.1097/00005768-199902000-00025 · Cited by 1,130 · see citing works
Design and Analysis of Group-Randomized Trials
Murray, D. M. (2007) · doi:10.1093/oso/9780195120363.001.0001 · Cited by 1,114 · see citing works
Evidence-Based Education Policies: Transforming Educational Practice and Research
Slavin, R. E. (2002) · Educational Researcher · doi:10.3102/0013189x031007015 · Cited by 1,107 · see citing works
Self-citationCausal Effects in Non-Experimental Studies: Re-Evaluating the Evaluation of Training Programs
Dehejia, R., & Wahba, S. (1998) · National Bureau of Economic Research · doi:10.3386/w6586 · Cited by 821 · see citing works
Comprehensive School Reform and Achievement: A Meta-Analysis
Borman, G. D., Hewes, G. M., Overman, L., & Brown, S. (2003) · Review of Educational Research · doi:10.3102/00346543073002125 · Cited by 797 · see citing works
Torgesen, J. K., Wagner, R. K., Rashotte, C. A., Rose, E., Lindamood, P. C., Conway, T. W., & Garvan, C. W. (1999) · Journal of Educational Psychology · doi:10.1037/0022-0663.91.4.579 · Cited by 719 · see citing works
Statistical analysis and optimal design for cluster randomized trials.
Raudenbush, S. W. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.2.173 · Cited by 636 · see citing works
Elbaum, B., Vaughn, S., Tejero Hughes, M., & Watson Moody, S. (2000) · Journal of Educational Psychology · doi:10.1037/0022-0663.92.4.605 · Cited by 564 · see citing works
School Science and Mathematics 101
Flick, L. B., & Lederman, N. G. (2002) · School Science and Mathematics · doi:10.1111/j.1949-8594.2002.tb18142.x · Cited by 447 · see citing works
Evidence matters: randomized trials in education research
(2003) · Choice Reviews Online · doi:10.5860/choice.40-3545 · Cited by 374 · see citing works
The Cooperative Elementary School: Effects on Students’ Achievement, Attitudes, and Social Relations
Stevens, R. J., & Slavin, R. E. (1995) · American Educational Research Journal · doi:10.3102/00028312032002321 · Cited by 367 · see citing works
Self-citationModeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
No Child Left Behind: A Desktop Reference.
Ohnemus, E. (2002) · Cited by 343 · see citing works
Cooperative Integrated Reading and Composition: Two Field Experiments
Stevens, R. J., Madden, N. A., Slavin, R. E., & Farnish, A. M. (1987) · Reading Research Quarterly · doi:10.2307/747701 · Cited by 290 · see citing works
Self-citationEffective Reading Programs for Middle and High Schools: A Best‐Evidence Synthesis
Slavin, R. E., Cheung, A., Groff, C., & Lake, C. (2008) · Reading Research Quarterly · doi:10.1598/rrq.43.3.4 · Cited by 283 · see citing works
Self-citationFinal Reading Outcomes of the National Randomized Field Trial of Success for All
Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A. M., Madden, N. A., & Chambers, B. (2007) · American Educational Research Journal · doi:10.3102/0002831207306743 · Cited by 218 · see citing works
Self-citationAn Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis
Dear, K., & Begg, C. (1992) · Statistical Science · doi:10.1214/ss/1177011363 · Cited by 202 · see citing works
(2003) · PsycEXTRA Dataset · doi:10.1037/e370412004-001 · Cited by 202 · see citing works
PUBLICATION BIAS: THE ACHILLES’ HEEL OF SYSTEMATIC REVIEWS?
Torgerson, C. (2006) · British Journal of Educational Studies · doi:10.1111/j.1467-8527.2006.00332.x · Cited by 178 · see citing works
Stevens, R. J., Slavin, R. E., & Farnish, A. M. (1991) · Journal of Educational Psychology · doi:10.1037/0022-0663.83.1.8 · Cited by 172 · see citing works
Self-citationToward Reform of Program Evaluation: Aims, Methods, and Institutional Arrangements.
Cain, P. S., & Cronbach, L. J. (1982) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2068404 · Cited by 167 · see citing works
Are Experiments the Only Option? A Look at Dropout Prevention Programs
Agodini, R., & Dynarski, M. (2004) · The Review of Economics and Statistics · doi:10.1162/003465304323023741 · Cited by 162 · see citing works
Stevens, R. J., & Slavin, R. E. (1995) · The Elementary School Journal · doi:10.1086/461801 · Cited by 159 · see citing works
Self-citation(2001) · Routledge eBooks · doi:10.4324/9781410600905 · Cited by 156 · see citing works
One million children: success for all
(2001) · Choice Reviews Online · doi:10.5860/choice.39-0454 · Cited by 152 · see citing works
Toward reform of program evaluation: Aims, methods, and institutional arrangements
Weiss, C. H. (1981) · Evaluation and Program Planning · doi:10.1016/0149-7189(81)90038-0 · Cited by 152 · see citing works
21st Century Education: A Reference Handbook
Good, T. L. (2008) · Sage eBooks · doi:10.4135/9781412964012 · Cited by 141 · see citing works
The Effectiveness of a Group Reading Instruction Program with Poor Readers in Multiple Grades
Rashotte, C. A., MacPhee, K., & Torgesen, J. K. (2001) · Learning Disability Quarterly · doi:10.2307/1511068 · Cited by 135 · see citing works
Baker, S., Gersten, R. M., & Keating, T. M. (2000) · Reading Research Quarterly · doi:10.1598/rrq.35.4.3 · Cited by 134 · see citing works
PET and the Pendulum: Faddism in Education and how to Stop It.
Slavin, R. E. (1989) · Phi Delta Kappan · Cited by 127 · see citing works
Self-citationGivens, G. H., Smith, D. D., & Tweedie, R. L. (1997) · Statistical Science · doi:10.1214/ss/1030037958 · Cited by 127 · see citing works
Advancing Scientific Research in Education
(2004) · doi:10.17226/11112 · Cited by 126 · see citing works
Heinsman, D. T., & Shadish, W. R. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.154 · Cited by 125 · see citing works
An Evaluation of Computer-Assisted Instruction in Phonological Awareness with below Average Readers
Barker, T. A., & Torgesen, J. K. (1995) · Journal of Educational Computing Research · doi:10.2190/th3m-btp7-jej5-jfnj · Cited by 100 · see citing works
Pastor, M., & Mollenkopf, J. H. (2016) · Cornell University Press eBooks · doi:10.7591/cornell/9781501702662.003.0010 · Cited by 94 · see citing works
Computer Administered Instruction in Phonological Awareness: Evaluation of the DaisyQuest Program.
Foster, K. C. (1994) · Journal of research and development in education · Cited by 78 · see citing works
Ysseldyke, J., Spicuzza, R., Kosciolek, S., Teelucksingh, E., Boys, C., & Lemkuil, A. (2003) · Journal of Education for Students Placed at Risk (JESPAR) · doi:10.1207/s15327671espr0802_4 · Cited by 67 · cited 1× · see citing works
“Numerous studies (e.g., Crawford & Snider, 2000; Van Dusen & Worthen, 1994; Ysseldyke et al., 2003) have used both national standardized tests and developer-made tests, and effect sizes are invariably much more positive on the latter measures.”
MethodNonexperimental Replications of Social Experiments: A Systematic Review
Glazerman, S., Levy, D. A., & Myers, D. (2014) · RePEc: Research Papers in Economics · Cited by 60 · see citing works
Torgerson, C. J. (2006) · Journal of Research in Reading · doi:10.1111/j.1467-9817.2006.00318.x · Cited by 59 · see citing works
Experiments for Educational Evaluation and Improvement
Borman, G. D. (2002) · Peabody Journal of Education · doi:10.1207/s15327930pje7704_2 · Cited by 55 · see citing works
Effective Mathematics Instruction the Importance of Curriculum
Crawford, D. B., & Snider, V. E. (2000) · Education and Treatment of Children · Cited by 38 · see citing works
Geometric Knowledge of Middle School Students in a Reform‐based Mathematics Curriculum
Carroll, W. M. (1998) · School Science and Mathematics · doi:10.1111/j.1949-8594.1998.tb17415.x · Cited by 34 · see citing works
Using Peer Response Groups with Limited English Proficient Writers
Prater, D. L., & Bermúdez, A. B. (1993) · Bilingual Research Journal · doi:10.1080/15235882.1993.10162650 · Cited by 30 · see citing works
Briggs, D. C. (2005) · Evaluation Review · doi:10.1177/0193841x04272555 · Cited by 22 · see citing works
Advancing Scientific Research in Education
Towne, L., Wise, L. L., & Winters, T. M. (2005) · PsycEXTRA Dataset · doi:10.1037/e401082005-001 · Cited by 19 · see citing works
Dusen, L. M. V., & Worthen, B. R. (1994) · International Journal of Educational Research · doi:10.1016/0883-0355(94)90020-5 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research
Wisniewski, B., Zierer, K., & Hattie, J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.03087 · Cited by 1,199 · see citing works
“We included studies with controlled designs as well as pre-post-test designs, and this became a moderator to investigate any differences related to design (Slavin, 2008).”
Schonert-Reichl, K. A., Oberle, E., Lawlor, M. S., Abbott, D., Thomson, K., Oberlander, T. F., & Diamond, A. (2015) · Developmental Psychology · doi:10.1037/a0038454 · Cited by 813 · see citing works
How Methodological Features Affect Effect Sizes in Education
Cheung, A. C. K., & Slavin, R. E. (2016) · Educational Researcher · doi:10.3102/0013189x16656615 · Cited by 529 · see citing works
Self-citationEmergency remote teaching in higher education: mapping the first global online semester
Bond, M., Bedenlier, S., Marín, V. I., & Händel, M. (2021) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-021-00282-x · Cited by 527 · see citing works
Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713493129 · Cited by 406 · see citing works
Schonert-Reichl, K. A., Smith, V., Zaidman-Zait, A., & Hertzman, C. (2012) · School Mental Health · doi:10.1007/s12310-011-9064-7 · Cited by 301 · see citing works
van de Pol, J., Volman, M., Oort, F., & Beishuizen, J. (2015) · Instructional Science · doi:10.1007/s11251-015-9351-z · Cited by 220 · see citing works
Parkhurst, J. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.4324/9781315675008 · Cited by 214 · see citing works
Hwang, G.-J., Hsu, T.-C., Lai, C.-L., & Hsueh, C.-J. (2017) · Computers & Education · doi:10.1016/j.compedu.2016.11.010 · Cited by 212 · see citing works
Strengthening the research base that informs STEM instructional improvement efforts: A meta-analysis
Lynch, K., Hill, H. C., Gonzalez, K. E., & Pollard, C. (2019) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373719849044 · Cited by 160 · see citing works
Cheang, R., Gillions, A., & Sparkes, E. (2019) · Journal of Child and Family Studies · doi:10.1007/s10826-019-01413-9 · Cited by 95 · see citing works
Creativity training programs in primary education: A systematic review and meta-analysis
Ruiz-del-Pino, B., Fernández-Martín, F. D., & Arco-Tirado, J. L. (2022) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2022.101172 · Cited by 48 · see citing works
<i>Comments on Slavin</i> : Synthesizing Causal Inferences
Briggs, D. C. (2008) · Educational Researcher · doi:10.3102/0013189x08314286 · Cited by 45 · see citing works
- Authors
- Slavin, R. E.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Slavin, Robert E.
- ORCID
- Slavin, Robert E.
- Venue
- Educational Researcher
- Published
- 2008-01
- Volume
- 37
- Issue
- 1
- Pages
- 5-14
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2157669419
- Semantic Scholar
- dad58f73341afafc19cb9681fb8aa591d8a9385d
- Cited by
- 560
- References
- 60
- Citations by year
- 2026: 21 · 2025: 14 · 2024: 15 · 2023: 27 · 2022: 34 · 2021: 41 · 2020: 29 · 2019: 36 · 2018: 17 · 2017: 25 · 2016: 42 · 2015: 36 · 2014: 27 · 2013: 36 · 2012: 41
- FWCI
- 45.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Researcher (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Social Sciences › Education › Educational Research and Methods
- Keywords
- evidence-based education, meta-analysis, research design, sample size, effect size, quality of evidence, educational program effectiveness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
