The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?
Open accessAbstract
There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K–12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Traditional educational interventions such as those aimed at improving overall teaching quality or providing targeted academic or practical support (Education Endowment Foundation, 2019) have met with some success, but tend to be expensive and/or resource intensive and in many cases have only fairly modest effects (Boulay et al., 2018;Lortie-Forgues & Inglis, 2019;Yeager & Walton, 2011).”
From Tackling Educational Inequalities with Social Psychology: Identities, Contexts, and Interventions (2021) · cites this paper as Lortie-Forgues & Inglis, 2019
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
(2000) · Cited by 9,198
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 4×
“Unfavorable findings from traditional research are less likely to be published (Rosenthal, 1979), and many researchers selectively report analyses and conduct unplanned analyses (John, Lowenstein, & Prelec, 2012).”
BackgroundResultInfluentialEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 3×
“Recent developments, collectively referred to as the ‘replication crisis’, suggest that the psychological literature is not as robust as previously imagined (e.g., Open Science Collaboration, 2015).”
BackgroundResultSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Ioannidis (2005) has shown that if a scientific field ignores the importance of replication, a situation can arise where “most published research findings are false”.”
BackgroundClinical Trials: A Practical Approach.
Hill, I. D., & Pocock, S. (1984) · Biometrics · doi:10.2307/2531186 · Cited by 2,918
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033
Self-Motivation for Academic Attainment: The Role of Self-Efficacy Beliefs and Personal Goal Setting
Zimmerman, B. J., Bandura, A., & Martinez-Pons, M. (1992) · American Educational Research Journal · doi:10.3102/00028312029003663 · Cited by 1,724
In defense of external invalidity.
Mook, D. G. (1983) · American Psychologist · doi:10.1037/0003-066x.38.4.379 · Cited by 1,284
Clinical trials: A practical approach
Daly, M. J. (1984) · Trends in Pharmacological Sciences · doi:10.1016/0165-6147(84)90515-7 · Cited by 1,057
Empirical Benchmarks for Interpreting Effect Sizes in Research
Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008) · Child Development Perspectives · doi:10.1111/j.1750-8606.2008.00061.x · Cited by 950
Bayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938 · cited 3×
“Bayes Factor For each trial we calculated a Bayes factor, following the method suggested by Dienes, Coulton and Heather (2017). This quantified how likely the data were under the null hypothesis compared to the alternative hypothesis, which was defined to be an effect size taken from a half normal distribution with me…”
BackgroundMethodInfluentialUnderstanding controlled trials: Why are randomised controlled trials important?
Sibbald, B., & Roland, M. (1998) · BMJ · doi:10.1136/bmj.316.7126.201 · Cited by 807
The SAGE Handbook of Social Research Methods
Alasuutari, P., Bickman, L., & Brannen, J. (2008) · SAGE Publications Ltd eBooks · doi:10.4135/9781446212165 · Cited by 759
Standardized or simple effect size: what should be reported?
Baguley, T. (2009) · British Journal of Psychology · doi:10.1348/000712608x377117 · Cited by 619
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · cited 1×
“This is worrying, as only 0.13% of articles in leading education journals report replication studies (Makel & Plucker, 2014).”
BackgroundInfluentialImproving Educational Research:Toward a More Useful, More Influential, and Better-Funded Enterprise
Burkhardt, H., & Schoenfeld, A. H. (2003) · Educational Researcher · doi:10.3102/0013189x032009003 · Cited by 583
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al‐Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 533
How Methodological Features Affect Effect Sizes in Education
Cheung, A. C. K., & Slavin, R. E. (2016) · Educational Researcher · doi:10.3102/0013189x16656615 · Cited by 529
Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W. (2008) · Journal of Research on Educational Effectiveness · doi:10.1080/19345740802400072 · Cited by 477
(2009) · Cited by 439
RCT = results confounded and trivial: the perils of grand educational experiments
Norman, G. R. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01586.x · Cited by 197
The Evaluation of Charter School Impacts: Final Report. NCEE 2010-4029.
Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010) · National Center for Education Evaluation and Regional Assistance · Cited by 160
Karlsson, P., & Bergmark, A. (2015) · Addiction · doi:10.1111/add.12799 · Cited by 148
Design Sensitivity in Criminal Justice Experiments
Weisburd, D., Petrosino, A., & Mason, G. (1993) · Crime and Justice · doi:10.1086/449216 · Cited by 124
Reading First Impact Study Final Report
Gamse, B., Jacob, R. T., Horst, M. A., Boulay, B. A., Associates, A., Unlu, F., Bozzi, L., Caswell, L., Rodger, C., Carter, W., et al. (2009) · Cited by 100
Evaluation of the DC Opportunity Scholarship Program: Impacts After Three Years
Wolf, P. J., Gutmann, B., Puma, M. J., Kisida, B., Rizzo, L., Eissa, N., & Silverberg, M. K. (2007) · Journal of the Arkansas Academy of Science · Cited by 70
Using Bayes factors to evaluate evidence for no effect: examples from the SIPS project
Dienes, Z. P., Coulton, S., & Heather, N. (2017) · Addiction · doi:10.1111/add.14002 · Cited by 66
Focusing on Mathematical Knowledge: The Impact of Content-Intensive Teacher Professional Development
Garet, M. S. (2016) · 2016 Fall Conference: The Role of Research in Making Government More Effective · Cited by 65
The Impact of the Measures of Academic Progress (MAP) Program on Student Reading Achievement
Cordray, D. S., Pion, G. M., Brandt, C., & Molefe, A. C. (2013) · Society for Research on Educational Effectiveness · Cited by 43
Clark, M. A., Chiang, H. S., Silva, T., McConnell, S., Sonnenfeld, K., Erbe, A., & Puma, M. J. (2013) · RePEc: Research Papers in Economics · Cited by 33
The Impact of Providing Performance Feedback to Teachers and Principals
Wayne, A. J. (2018) · 2018 APPAM Fall Research Conference: Evidence for Action: Encouraging Innovation and Improvement · Cited by 17
Mathematics Mastery: Primary Evaluation Report.
Vignoles, A. F., Jerrim, J., & Cowan, R. (2015) · Cited by 14
James‐Burdumy, S., Deke, J., Lugo‐Gil, J., Carey, N., Hershey, A. M., Gersten, R. M., Newman-Gonchar, R., Dimino, J., Haymond, K. S., & Faddis, B. (2010) · RePEc: Research Papers in Economics · Cited by 10
Evaluation of Principles and Practices
Dewey, J., Mayhew, K. C., & Edwards, A. C. (2017) · Routledge eBooks · doi:10.4324/9781315131733-24 · Cited by 9
Increasing Pupil Motivation: Evaluation Report and Executive Summary.
Sibieta, L., Greaves, E., & Sianesi, B. (2014) · Cited by 5
Research works printed but not linked: 5
- Education Endowment Foundation, 2015, Annual report 2014/15
- Education Endowment Foundation, 2015, EEF guidance on cost evaluation
- Education Endowment Foundation, 2016, Classification of the security of findings from EEF evaluations
- Education Endowment Foundation, 2017, EEF standards for independent evaluation panel members
- Jeffreys H., 1961, Theory of probability
Cited by
Sorted by citations
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
Post Qualitative Inquiry, the Refusal of Method, and the Risk of the New
Pierre, E. A. S. (2019) · Qualitative Inquiry · doi:10.1177/1077800419863005 · Cited by 313
Beyond small, medium, or large: points of consideration when interpreting effect sizes
Bakker, A., Cai, J., English, L., Kaiser, G., Mesa, V., & Van Dooren, W. (2019) · Educational Studies in Mathematics · doi:10.1007/s10649-019-09908-4 · Cited by 302
Scaling up behavioral science interventions in online education
Kizilcec, R. F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., Turkay, S., Williams, J. J., & Tingley, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1921417117 · Cited by 208
Achievement Gaps in the Wake of COVID-19
Bailey, D. H., Duncan, G. J., Murnane, R. J., & Au Yeung, N. (2021) · Educational Researcher · doi:10.3102/0013189x211011237 · Cited by 134
Metadata
- Authors
- Lortie-Forgues, H., & Inglis, M.
- Affiliations
- University of York, UK
- Loughborough University (GB)
- ORCID
- Inglis, Matthew
- Venue
- Educational Researcher
- Published
- 2019-03-11
- Volume
- 48
- Issue
- 3
- Pages
- 158-166
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2919428740
- Cited by
- 234
- References
- 44
- Citations by year
- 2026: 19 · 2025: 20 · 2024: 27 · 2023: 19 · 2022: 64 · 2021: 39 · 2020: 30 · 2019: 16
- FWCI
- 24.93
- Type
- journal article
- Language
- en
- Open access
- green
- orcid.org
- Locations
- Educational Researcher (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Loughborough University Institutional Repository (Loughborough University) (submittedVersion, cc-by-nc-nd)
- Loughborough University Research Repository (Loughborough University) (submittedVersion, cc-by-nc-nd)
- Funders
- university of york
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Education › Educational Research and Methods
- Keywords
- educational intervention, effect size, confidence intervals, Bayes factor, statistical power, Education Endowment Foundation, evidence-based education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
