The Relationship Between Sample Sizes and Effect Sizes in Systematic Reviews in Education
The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Research in fields other than education has found that studies with small sample sizes tend to have larger effect sizes than those with large samples. This article examines the relationship between sample size and effect size in education. It analyzes data from 185 studies of elementary and secondary mathematics programs that met the standards of the Best Evidence Encyclopedia. As predicted, there was a significant negative correlation between sample size and effect size. The differences in effect sizes between small and large experiments were much greater than those between randomized and matched experiments. Explanations for the effects of sample size on effect size are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2000) · Cited by 9,198
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Kjaergard, L. L., Villumsen, J., & Gluud, C. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,283
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · cited 3×
“The main reason advanced by methodologists for effects of small samples in experimental studies in medicine is that publication bias is more serious in small sample research than in studies with large samples (Givens et al., 1997; Rothstein, Sutton, & Borenstein, 2005; Sterne et al., 2000; Taylor & Tweedy, 1998).”
BackgroundThe review of educational research.
(1931) · Journal of Educational Psychology · doi:10.1037/h0066448 · Cited by 1,562
Comprehensive School Reform and Achievement: A Meta-Analysis
Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003) · Review of Educational Research · doi:10.3102/00346543073002125 · Cited by 795
Torgesen, J. K., Wagner, R. K., Rashotte, C. A., Rose, E., Lindamood, P., Conway, T., & Garvan, C. (1999) · Journal of Educational Psychology · doi:10.1037//0022-0663.91.4.579 · Cited by 724
Cook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604 · cited 4×
“As noted by Cook et al. (2008), this does not mean that random assignment is unimportant, but it does imply that factors other than random assignment also need to be taken into account. In particular, the present findings argue against the policy of the WWC (2008b) of placing extraordinary value on randomized experime…”
BackgroundResultInfluentialSlavin, R. E. (2008) · Educational Researcher · doi:10.3102/0013189x08314117 · Cited by 560 · cited 10×
“For example, sample size weighting was used by Borman, Hewes, Overman, and Brown (2003) in a review of research on comprehensive school reform models and by Slavin et al. (2008); Slavin, Lake, and Groff (2009); and Slavin, Lake, Chambers, et al.”
BackgroundMethodSelf-citationEffective Programs in Elementary Mathematics: A Best-Evidence Synthesis
Slavin, R. E., & Lake, C. (2008) · Review of Educational Research · doi:10.3102/0034654308317473 · Cited by 407
Self-citation
Cited by
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
Mind-Set Interventions Are a Scalable Treatment for Academic Underachievement
Paunesku, D., Walton, G. M., Romero, C., Smith, E. N., Yeager, D. S., & Dweck, C. S. (2015) · Psychological Science · doi:10.1177/0956797615571017 · Cited by 981
Cheung, A. C. K., & Slavin, R. E. (2013) · Educational Research Review · doi:10.1016/j.edurev.2013.01.001 · Cited by 792
Self-citationBai, S., Hew, K. F., & Huang, B. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100322 · Cited by 994
Godwin, A., Potvin, G., Hazari, Z., & Lock, R. (2016) · Journal of Engineering Education · doi:10.1002/jee.20118 · Cited by 574
Hillmayr, D., Ziernwald, L., Reinhold, F., Hofer, S. I., & Reiss, K. M. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103897 · Cited by 574
“Empirically, studies with smaller sample sizes produce larger effects (Chen et al., 2018; Cheung & Slavin, 2013; Slavin and Smith, 2009).”
BackgroundA meta-analysis of online trust relationships in e-commerce.
Kim, Y., & Peterson, R. A. (2017) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2017.01.001 · Cited by 541
How Methodological Features Affect Effect Sizes in Education
Cheung, A. C. K., & Slavin, R. E. (2016) · Educational Researcher · doi:10.3102/0013189x16656615 · Cited by 529
Self-citationWhen and why defaults influence decisions: a meta-analysis of default effects
JACHIMOWICZ, J. M., DUNCAN, S., WEBER, E. U., & JOHNSON, E. J. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2018.43 · Cited by 481
Graham, C. D., Gouick, J., Krahé, C., & Gillanders, D. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.04.009 · Cited by 476
Chen, J., Wang, M., Kirschner, P. A., & Tsai, C.-C. (2018) · Review of Educational Research · doi:10.3102/0034654318791584 · Cited by 450
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399
Differentiated Instruction in Secondary Education: A Systematic Review of Research Evidence
Smale-Jacobse, A. E., Meijer, A., Helms-Lorenz, M., & Maulana, R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02366 · Cited by 404
CEC’s Standards for Classifying the Evidence Base of Practices in Special Education
Cook, B. G., Buysse, V., Klingner, J., Landrum, T. J., McWilliam, R. A., Tankersley, M., & Test, D. W. (2015) · Remedial and Special Education · doi:10.1177/0741932514557271 · Cited by 361
Effective programs for struggling readers: A best-evidence synthesis
Slavin, R. E., Lake, C., Davis, S., & Madden, N. A. (2011) · Educational Research Review · doi:10.1016/j.edurev.2010.07.002 · Cited by 353
Self-citationHendriks, T., Schotanus-Dijkstra, M., Hassankhan, A., de Jong, J., & Bohlmeijer, E. (2020) · Journal of Happiness Studies · doi:10.1007/s10902-019-00082-1 · Cited by 317
du Plooy, E., Casteleijn, D., & Franzsen, D. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e39630 · Cited by 310
How features of educational technology applications affect student reading outcomes: A meta-analysis
Cheung, A. C. K., & Slavin, R. E. (2012) · Educational Research Review · doi:10.1016/j.edurev.2012.05.002 · Cited by 307
Self-citationXu, Z., Zhao, Y., Liew, J., Zhou, X., & Kogut, A. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100510 · Cited by 290
Guest, E., Costa, B., Williamson, H., Meyrick, J., Halliwell, E., & Harcourt, D. (2019) · Body Image · doi:10.1016/j.bodyim.2019.04.002 · Cited by 184
“This suggests that being part of a group may enhance the effectiveness of interventions; however, it is important to note that Ahmadi et al. (2017) obtained a small sample size (n = 24), which may be associated with the very large effect sizes achieved from their intervention (Slavin & Smith, 2009).”
BackgroundGallegos, A. M., Crean, H. F., Pigeon, W. R., & Heffner, K. L. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.10.004 · Cited by 181
“Previous meta-analytic reviews have noted that small sample size is associated with higher ESs (Ioannidis, Cappelleri, & Lau, 1998; Slavin & Smith, 2009; Sterne, Gavaghan, & Egger, 2000).”
A Scoping Review of Digital Game-Based Technology on English Language Learning
Xu, Z., Chen, Z., Eutsler, L., Geng, Z., & Kogut, A. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-019-09702-2 · Cited by 162
Effects of digital games on student motivation in mathematics: A meta‐analysis in K‐12
Fadda, D., Pellegrini, M., Vivanet, G., & Zandonella Callegher, C. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12618 · Cited by 135
Schmid, R. F., Borokhovski, E., Bernard, R. M., Pickup, D. I., & Abrami, P. C. (2023) · Computers and Education Open · doi:10.1016/j.caeo.2023.100142 · Cited by 70
What is needed to eradicate the depression epidemic, and why
Ormel, J., Cuijpers, P., Jorm, A., & Schoevers, R. A. (2020) · Mental Health & Prevention · doi:10.1016/j.mhp.2019.200177 · Cited by 67
A framework for learning from null results
Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A. (2019) · Educational Researcher · doi:10.3102/0013189x19891955 · Cited by 65
“However, as has been shown in education and other fields, sample size has a strong negative correlation with effect size (Slavin & Smith, 2009).”
BackgroundOn the value of data-based decision making in education: The evidence from six intervention studies
Visscher, A. J. (2021) · Studies in Educational Evaluation · doi:10.1016/j.stueduc.2020.100899 · Cited by 61
Chae, Y., Lee, S., & Kim, Y. (2019) · Asian Journal of Communication · doi:10.1080/01292986.2018.1499121 · Cited by 55
Atefi, G. L., de Vugt, M. E., van Knippenberg, R. J. M., Levin, M. E., Verhey, F. R. J., & Bartels, S. L. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2023.102341 · Cited by 33
Seo, C., Kim, B., & Kruis, N. E. (2021) · International Criminal Justice Review · doi:10.1177/1057567720979184 · Cited by 24
Stanyer, E. C., Baniqued, P. D. E., Awais, M., Kouara, L., Davies, A. G., Killan, E. C., & Mushtaq, F. (2022) · Journal of Sleep Research · doi:10.1111/jsr.13385 · Cited by 21
RetractedConsumer confusion: A meta-analytic review
Shahid, S., Maseeh, H. I., Jebarajakirthy, C., Kumar Sharma, B., Saha, R., & Gupta, S. (2025) · Journal of Business Research · doi:10.1016/j.jbusres.2025.115294 · Cited by 13
Metadata
- Authors
- Slavin, R., & Smith, D.
- Venue
- Educational Evaluation and Policy Analysis
- Published
- 2009-12
- Volume
- 31
- Issue
- 4
- Pages
- 500-506
- ISSN
- 0162-3737 (ISSN-L)
- 1935-1062
- OpenAlex
- W2169029212
- Cited by
- 360
- References
- 20
- FWCI
- 0.53
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.