The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?
Open accessAbstract
There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K–12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Traditional educational interventions such as those aimed at improving overall teaching quality or providing targeted academic or practical support (Education Endowment Foundation, 2019) have met with some success, but tend to be expensive and/or resource intensive and in many cases have only fairly modest effects (Boulay et al., 2018;Lortie-Forgues & Inglis, 2019;Yeager & Walton, 2011).”
From Tackling Educational Inequalities with Social Psychology: Identities, Contexts, and Interventions (2021) · cites this paper as Lortie-Forgues & Inglis, 2019
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
(2000) · Cited by 9,198
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 4×
“Unfavorable findings from traditional research are less likely to be published (Rosenthal, 1979), and many researchers selectively report analyses and conduct unplanned analyses (John, Lowenstein, & Prelec, 2012).”
BackgroundResultInfluentialEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 3×
“Recent developments, collectively referred to as the ‘replication crisis’, suggest that the psychological literature is not as robust as previously imagined (e.g., Open Science Collaboration, 2015).”
BackgroundResultSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Research works printed but not linked: 5
- Education Endowment Foundation, 2015, Annual report 2014/15
- Education Endowment Foundation, 2015, EEF guidance on cost evaluation
- Education Endowment Foundation, 2016, Classification of the security of findings from EEF evaluations
- Education Endowment Foundation, 2017, EEF standards for independent evaluation panel members
- Jeffreys H., 1961, Theory of probability
Cited by
Sorted by citations
Items 201–234 of 234
Smith, B., Morris, S. P., & Armitage, H. (2022) · Research Papers in Education · doi:10.1080/02671522.2022.2065522 · Cited by 1
Baker, G. (2021) · Open Book Publishers · doi:10.11647/obp.0243.06
Baker, G. (2021) · Open Book Publishers · doi:10.11647/obp.0243.05
Baker, G. (2021) · Open Book Publishers · doi:10.11647/obp.0243.08
Foundations: Is Measurement the Enemy of Creativity?
Barbetta, G. (2022) · Nonprofit and civil society studies · doi:10.1007/978-3-030-98008-5_12
Behandlingskontrast i eksperimenter innen velferd og utdanning
Malmberg-Heimonen, I. E., & Tøge, A. G. (2022) · Tidsskrift for samfunnsforskning · doi:10.18261/tfs.63.4.3
Schools as research-informed professional learning organizations
Godfrey, D. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-818630-5.05045-4
Browitt, A., Croll, N., Hedge-Holmes, K., & Henry, K. (2023) · Widening Participation and Lifelong Learning · doi:10.5456/wpll.24.3.10
Reliability Issues with At-Home Assessment During the COVID-19 Pandemic
Shamir, H., Yoder, E., & Pocklington, D. (2023) · Smart innovation, systems and technologies · doi:10.1007/978-981-99-2993-1_9
Innovation and Change Within Education
Tymms, P. B., & Bolden, D. S. (2023) · Palgrave studies in creativity and culture · doi:10.1007/978-3-031-28206-5_3
MacKay, J. R. D., & Wu, S. C. (2023) · Wiley eBooks · doi:10.1002/9781119852865.ch17 · Cited by 3
Olson, H. A., Ozernov‐Palchik, O., Arechiga, X. M., & Gabrieli, J. D. E. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/w89ea
Brunner, M., Stallasch, S. E., Artelt, C., & Lüdtke, O. (2024) · doi:10.31234/osf.io/dkw42
ManyClasses 2: The Effects of Prequestions on Media Interactions and LearningPreprint
Motz, B. A., Chinni, A., de Leeuw, J. R., Jankowski, H. E., Aggarwal, A., Amato, M., Berlin, K. L., Britten, K., Brown, A., Cerchiaro, M. L., et al. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/3xbma · Cited by 1
Alcock, L., & Inglis, M. (2025) · Research in mathematics education · doi:10.1007/978-3-031-85004-2_16
Self-citationFaherty, R., & Quille, K. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-92608-2_40
Beyond anecdotes: Incorporating qualitative methods with rigor in economic education research
Birdi, A., Jennings, A. B., & Spielmann, C. (2025) · The Journal of Economic Education · doi:10.1080/00220485.2025.2583079
Cheung, J. J. H., & Merchant, A. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02577-4
Building the Evidence Base to Reduce Educational Poverty and Inequalities
Coldwell, M., & Demack, S. (2025) · Springer eBooks · doi:10.1007/978-3-658-44698-7_31-1
Sánchez-López, R., & Hernando, E. (2026) · Football Studies · doi:10.1016/j.footst.2026.100025
Pedagogisk forskning trenger store utvalg
Nordmo, M., & Myrene, K. (2026) · Norsk pedagogisk tidsskrift · doi:10.18261/npt.110.1.7
Olson, H. A., Ozernov‐Palchik, O., Arechiga, X. M., & Gabrieli, J. D. E. (2026) · Developmental Science · doi:10.1111/desc.70159
Kalliyil, M., & Sahoo, S. (2026) · Journal of Development Economics · doi:10.1016/j.jdeveco.2026.103790
La méthode de la "classe puzzle" est-elle efficace pour améliorer l'apprentissage ?
Stanczak, A. (2020) · theses.fr (ABES) · doi:10.70675/7c1ef139zd9e9z444bzbc73z7e6e82534c17
Reply to: Why can’t epidemiology be automated (yet)? Perhaps it shouldn’t
Bann, D., Lowther, E., Wright, L., & Kovalchuk, Y. (2026) · International Journal of Epidemiology · doi:10.1093/ije/dyag066
Ensayos controlados aleatorizados (rcts) en las ciencias de la educación en latinoamérica
Puente, J. H. (2024) · International Journal of Innovation and Pedagogy · doi:10.64325/rhh00q09
Rosa, N. D. L., Daley, J. S., & Balcetis, E. (2026) · Self and Identity · doi:10.1080/15298868.2026.2682150
Wu, R., & Wen, J. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1811130
Kühlwein, F., Merk, S., Schneider, J., & Schmidt, K. (2026) · Learning and Instruction · doi:10.1016/j.learninstruc.2026.102427
Building the Evidence Base to Reduce Educational Poverty and Inequalities
Coldwell, M., & Demack, S. (2026) · Springer eBooks · doi:10.1007/978-3-658-48108-7_31
Phạm, L. (2026) · Evaluation Journal of Australasia · doi:10.1177/1035719x261476074
Farran, E. K., Alexiou, S., Danielsson, H., Orefice, C., Cristescu, L., Pellicano, E., Herwegen, J. V., & Scerif, G. (2026) · British Journal of Developmental Psychology · doi:10.1111/bjdp.70065
A meta-analysis on educational programming tool effectiveness
Weißenrieder, F., Schmiegel, D., Altın, R., Lenzer, S., & Mühling, A. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101369
Metadata
- Authors
- Lortie-Forgues, H., & Inglis, M.
- Affiliations
- University of York, UK
- Loughborough University (GB)
- ORCID
- Inglis, Matthew
- Venue
- Educational Researcher
- Published
- 2019-03-11
- Volume
- 48
- Issue
- 3
- Pages
- 158-166
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2919428740
- Cited by
- 234
- References
- 44
- Citations by year
- 2026: 19 · 2025: 20 · 2024: 27 · 2023: 19 · 2022: 64 · 2021: 39 · 2020: 30 · 2019: 16
- FWCI
- 24.93
- Type
- journal article
- Language
- en
- Open access
- green
- orcid.org
- Locations
- Educational Researcher (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Loughborough University Institutional Repository (Loughborough University) (submittedVersion, cc-by-nc-nd)
- Loughborough University Research Repository (Loughborough University) (submittedVersion, cc-by-nc-nd)
- Funders
- university of york
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Education › Educational Research and Methods
- Keywords
- educational intervention, effect size, confidence intervals, Bayes factor, statistical power, Education Endowment Foundation, evidence-based education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
