The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index.
Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?
Open accessAbstract
There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K–12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Traditional educational interventions such as those aimed at improving overall teaching quality or providing targeted academic or practical support (Education Endowment Foundation, 2019) have met with some success, but tend to be expensive and/or resource intensive and in many cases have only fairly modest effects (Boulay et al., 2018;Lortie-Forgues & Inglis, 2019;Yeager & Walton, 2011).”
From Tackling Educational Inequalities with Social Psychology: Identities, Contexts, and Interventions (2021) · cites this paper as Lortie-Forgues & Inglis, 2019
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
(2000) · Cited by 9,198
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 4×
“Unfavorable findings from traditional research are less likely to be published (Rosenthal, 1979), and many researchers selectively report analyses and conduct unplanned analyses (John, Lowenstein, & Prelec, 2012).”
BackgroundResultInfluentialEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 3×
“Recent developments, collectively referred to as the ‘replication crisis’, suggest that the psychological literature is not as robust as previously imagined (e.g., Open Science Collaboration, 2015).”
BackgroundResultSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Research works printed but not linked: 5
- Education Endowment Foundation, 2015, Annual report 2014/15
- Education Endowment Foundation, 2015, EEF guidance on cost evaluation
- Education Endowment Foundation, 2016, Classification of the security of findings from EEF evaluations
- Education Endowment Foundation, 2017, EEF standards for independent evaluation panel members
- Jeffreys H., 1961, Theory of probability
Cited by
Sorted by citations
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281
Post Qualitative Inquiry, the Refusal of Method, and the Risk of the New
Pierre, E. A. S. (2019) · Qualitative Inquiry · doi:10.1177/1077800419863005 · Cited by 313
Beyond small, medium, or large: points of consideration when interpreting effect sizes
Bakker, A., Cai, J., English, L., Kaiser, G., Mesa, V., & Van Dooren, W. (2019) · Educational Studies in Mathematics · doi:10.1007/s10649-019-09908-4 · Cited by 302
Scaling up behavioral science interventions in online education
Kizilcec, R. F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., Turkay, S., Williams, J. J., & Tingley, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1921417117 · Cited by 208
Achievement Gaps in the Wake of COVID-19
Bailey, D. H., Duncan, G. J., Murnane, R. J., & Au Yeung, N. (2021) · Educational Researcher · doi:10.3102/0013189x211011237 · Cited by 134
Metadata
- Authors
- Lortie-Forgues, H., & Inglis, M.
- Affiliations
- University of York, UK
- Loughborough University (GB)
- ORCID
- Inglis, Matthew
- Venue
- Educational Researcher
- Published
- 2019-03-11
- Volume
- 48
- Issue
- 3
- Pages
- 158-166
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2919428740
- Cited by
- 234
- References
- 44
- Citations by year
- 2026: 19 · 2025: 20 · 2024: 27 · 2023: 19 · 2022: 64 · 2021: 39 · 2020: 30 · 2019: 16
- FWCI
- 24.93
- Type
- journal article
- Language
- en
- Open access
- green
- orcid.org
- Locations
- Educational Researcher (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Loughborough University Institutional Repository (Loughborough University) (submittedVersion, cc-by-nc-nd)
- Loughborough University Research Repository (Loughborough University) (submittedVersion, cc-by-nc-nd)
- Funders
- university of york
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Education › Educational Research and Methods
- Keywords
- educational intervention, effect size, confidence intervals, Bayes factor, statistical power, Education Endowment Foundation, evidence-based education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
