Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?
Open accessAbstract
There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational Evaluation and Regional Assistance, which evaluated interventions aimed at improving academic achievement in K–12 (141 RCTs; 1,222,024 students). The mean effect size was 0.06 standard deviations. These sat within relatively large confidence intervals (mean width = 0.30 SDs), which meant that the results were often uninformative (the median Bayes factor was 0.56). We argue that our field needs, as a priority, to understand why educational RCTs often find small and uninformative effects.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Traditional educational interventions have met with some success but tend to be expensive and/or resource intensive and often have only modest effects.
“Traditional educational interventions such as those aimed at improving overall teaching quality or providing targeted academic or practical support (Education Endowment Foundation, 2019) have met with some success, but tend to be expensive and/or resource intensive and in many cases have only fairly modest effects (Boulay et al., 2018;Lortie-Forgues & Inglis, 2019;Yeager & Walton, 2011).”
From Tackling Educational Inequalities with Social Psychology: Identities, Contexts, and Interventions (2021) · cites this paper as Lortie-Forgues & Inglis, 2019
The Effect Sizes section reports that 91 of 141 trials found effects above zero, but the mean effect was a modest 0.06 standard deviations. The paper calls large-scale RCTs expensive; it does not establish that the interventions themselves tend to be expensive or resource intensive. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799 · see citing works
(2000) · Cited by 9,198 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 4× · see citing works
“Unfavorable findings from traditional research are less likely to be published (Rosenthal, 1979), and many researchers selectively report analyses and conduct unplanned analyses (John, Lowenstein, & Prelec, 2012).”
BackgroundResultInfluentialEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 3× · see citing works
“Recent developments, collectively referred to as the ‘replication crisis’, suggest that the psychological literature is not as robust as previously imagined (e.g., Open Science Collaboration, 2015).”
BackgroundResultSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Clinical Trials: A Practical Approach.
Hill, I. D., & Pocock, S. (1984) · Biometrics · doi:10.2307/2531186 · Cited by 2,918 · see citing works
Self-Motivation for Academic Attainment: The Role of Self-Efficacy Beliefs and Personal Goal Setting
Zimmerman, B. J., Bandura, A., & Martinez-Pons, M. (1992) · American Educational Research Journal · doi:10.2307/1163261 · Cited by 2,709 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · see citing works
Self-Motivation for Academic Attainment: The Role of Self-Efficacy Beliefs and Personal Goal Setting
Zimmerman, B. J., Bandura, A., & Martinez-Pons, M. (1992) · American Educational Research Journal · doi:10.3102/00028312029003663 · Cited by 1,724 · see citing works
In defense of external invalidity.
Mook, D. G. (1983) · American Psychologist · doi:10.1037/0003-066x.38.4.379 · Cited by 1,221 · see citing works
Clinical trials: A practical approach
Daly, M. J. (1984) · Trends in Pharmacological Sciences · doi:10.1016/0165-6147(84)90515-7 · Cited by 1,057 · see citing works
Empirical Benchmarks for Interpreting Effect Sizes in Research
Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008) · Child Development Perspectives · doi:10.1111/j.1750-8606.2008.00061.x · Cited by 950 · see citing works
Bayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938 · see citing works
Understanding controlled trials: Why are randomised controlled trials important?
Sibbald, B., & Roland, M. (1998) · BMJ · doi:10.1136/bmj.316.7126.201 · Cited by 807 · see citing works
The SAGE Handbook of Social Research Methods
Alasuutari, P., Bickman, L., & Brannen, J. (2008) · SAGE Publications Ltd eBooks · doi:10.4135/9781446212165 · Cited by 759 · see citing works
Standardized or simple effect size: what should be reported?
Baguley, T. (2009) · British Journal of Psychology · doi:10.1348/000712608x377117 · Cited by 619 · see citing works
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · cited 1× · see citing works
“This is worrying, as only 0.13% of articles in leading education journals report replication studies (Makel & Plucker, 2014).”
BackgroundInfluentialImproving Educational Research:Toward a More Useful, More Influential, and Better-Funded Enterprise
Burkhardt, H., & Schoenfeld, A. H. (2003) · Educational Researcher · doi:10.3102/0013189x032009003 · Cited by 563 · see citing works
How Methodological Features Affect Effect Sizes in Education
Cheung, A. C. K., & Slavin, R. E. (2016) · Educational Researcher · doi:10.3102/0013189x16656615 · Cited by 529 · see citing works
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al‐Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 504 · see citing works
Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W. (2008) · Journal of Research on Educational Effectiveness · doi:10.1080/19345740802400072 · Cited by 477 · see citing works
(2009) · Cited by 439 · see citing works
RCT = results confounded and trivial: the perils of grand educational experiments
Norman, G. R. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01586.x · Cited by 197 · see citing works
The Evaluation of Charter School Impacts: Final Report. NCEE 2010-4029.
Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010) · National Center for Education Evaluation and Regional Assistance · Cited by 160 · see citing works
Karlsson, P., & Bergmark, A. (2015) · Addiction · doi:10.1111/add.12799 · Cited by 148 · see citing works
Design Sensitivity in Criminal Justice Experiments
Weisburd, D., Petrosino, A., & Mason, G. (1993) · Crime and Justice · doi:10.1086/449216 · Cited by 124 · see citing works
Reading First Impact Study Final Report
Gamse, B., Jacob, R. T., Horst, M. A., Boulay, B. A., Associates, A., Unlu, F., Bozzi, L., Caswell, L., Rodger, C., Carter, W., et al. (2009) · Cited by 101 · see citing works
Evaluation of the DC Opportunity Scholarship Program: Impacts After Three Years
Wolf, P. J., Gutmann, B., Puma, M. J., Kisida, B., Rizzo, L., Eissa, N., & Silverberg, M. K. (2007) · Journal of the Arkansas Academy of Science · Cited by 70 · see citing works
Focusing on Mathematical Knowledge: The Impact of Content-Intensive Teacher Professional Development
Garet, M. S. (2016) · 2016 Fall Conference: The Role of Research in Making Government More Effective · Cited by 65 · see citing works
Using Bayes factors to evaluate evidence for no effect: examples from the SIPS project
Dienes, Z. P., Coulton, S., & Heather, N. (2017) · Addiction · doi:10.1111/add.14002 · Cited by 65 · see citing works
The Impact of the Measures of Academic Progress (MAP) Program on Student Reading Achievement
Cordray, D. S., Pion, G. M., Brandt, C., & Molefe, A. C. (2013) · Society for Research on Educational Effectiveness · Cited by 43 · see citing works
Clark, M. A., Chiang, H. S., Silva, T., McConnell, S., Sonnenfeld, K., Erbe, A., & Puma, M. J. (2013) · RePEc: Research Papers in Economics · Cited by 33 · see citing works
The Impact of Providing Performance Feedback to Teachers and Principals
Wayne, A. J. (2018) · 2018 APPAM Fall Research Conference: <i>Evidence for Action: Encouraging Innovation and Improvement</i> · Cited by 17 · see citing works
Mathematics Mastery: Primary Evaluation Report.
Vignoles, A. F., Jerrim, J., & Cowan, R. (2015) · Cited by 14 · see citing works
James‐Burdumy, S., Deke, J., Lugo‐Gil, J., Carey, N., Hershey, A. M., Gersten, R. M., Newman-Gonchar, R., Dimino, J., Haymond, K. S., & Faddis, B. (2010) · RePEc: Research Papers in Economics · Cited by 10 · see citing works
Evaluation of Principles and Practices
Dewey, J., Mayhew, K. C., & Edwards, A. C. (2017) · Routledge eBooks · doi:10.4324/9781315131733-24 · Cited by 9 · see citing works
Increasing Pupil Motivation: Evaluation Report and Executive Summary.
Sibieta, L., Greaves, E., & Sianesi, B. (2014) · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Cheung, A. & Slavin, R. E. (2016). How methodological features of research studies affect effect sizes. Educational Researcher, 45 (5), 283-292.
- Jeffreys, H. (1961). Theory of probability (3rd ed.). Oxford, UK: Oxford University Press
Cited by
Sorted by citations
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281 · see citing works
Post Qualitative Inquiry, the Refusal of Method, and the Risk of the New
Pierre, E. A. S. (2019) · Qualitative Inquiry · doi:10.1177/1077800419863005 · Cited by 313 · see citing works
Beyond small, medium, or large: points of consideration when interpreting effect sizes
Bakker, A., Cai, J., English, L., Kaiser, G., Mesa, V., & Van Dooren, W. (2019) · Educational Studies in Mathematics · doi:10.1007/s10649-019-09908-4 · Cited by 302 · see citing works
Scaling up behavioral science interventions in online education
Kizilcec, R. F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., Turkay, S., Williams, J. J., & Tingley, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1921417117 · Cited by 208 · see citing works
Achievement Gaps in the Wake of COVID-19
Bailey, D. H., Duncan, G. J., Murnane, R. J., & Au Yeung, N. (2021) · Educational Researcher · doi:10.3102/0013189x211011237 · Cited by 134 · see citing works
Tackling Educational Inequalities with Social Psychology: Identities, Contexts, and Interventions
Easterbrook, M. J., & Hadden, I. R. (2021) · Social Issues and Policy Review · doi:10.1111/sipr.12070 · Cited by 110 · see citing works
Building Content Knowledge to Boost Comprehension in the Primary Grades
Cabell, S. Q., & Hwang, H. (2020) · Reading Research Quarterly · doi:10.1002/rrq.338 · Cited by 104 · see citing works
Both Questionable and Open Research Practices Are Prevalent in Education Research
Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021) · Educational Researcher · doi:10.3102/0013189x211001356 · Cited by 92 · see citing works
Gore, J. M., Miller, A., Fray, L., Harris, J., & Prieto, E. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103297 · Cited by 90 · see citing works
Educational research: an unorthodox introduction
Thomas, G. (2020) · Educational Review · doi:10.1080/00131911.2020.1813944 · Cited by 84 · see citing works
Training spatial cognition enhances mathematical learning in a randomized study of 17,000 children
Judd, N., & Klingberg, T. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01118-4 · Cited by 77 · see citing works
Relevance of Educational Research: An Ontological Conceptualization
Akkerman, S. F., Bakker, A., & Penuel, W. R. (2021) · Educational Researcher · doi:10.3102/0013189x211028239 · Cited by 80 · see citing works
Professional learning networks: a conceptual model and research opportunities
Poortman, C. L., Brown, C., & Schildkamp, K. (2021) · Educational Research · doi:10.1080/00131881.2021.1985398 · Cited by 74 · see citing works
The Role of Context in Educational RCT Findings: A Call to Redefine “Evidence-Based Practice”
Kaplan, A., Cromley, J. G., Perez, T., Dai, T., Mara, K. R., & Balsai, M. J. (2020) · Educational Researcher · doi:10.3102/0013189x20921862 · Cited by 64 · see citing works
Spatial thinking as the missing piece in mathematics curricula
Gilligan-Lee, K. A., Hawes, Z. C. K., & Mix, K. S. (2022) · npj Science of Learning · doi:10.1038/s41539-022-00128-9 · Cited by 64 · see citing works
Ferrero, M., Vadillo, M. A., & León, S. P. (2021) · Intelligence · doi:10.1016/j.intell.2021.101566 · Cited by 79 · see citing works
Baker, G. (2021) · Open Book Publishers · doi:10.11647/obp.0243
Growth-Mindset Intervention Delivered by Teachers Boosts Achievement in Early Adolescence
Porter, T., Molina, D. C., Cimpian, A., Roberts, S., Fredericks, A. C., Blackwell, L., & Trzesniewski, K. (2022) · Psychology Science · doi:10.1177/09567976211061109 · Cited by 49 · see citing works
Outhwaite, L. A., Gulliford, A., & Pitchford, N. J. (2019) · International Journal of Research & Method in Education · doi:10.1080/1743727x.2019.1657081 · Cited by 62 · see citing works
Neglect of publication bias compromises meta-analyses of educational research
Ropovik, I., Adamkovic, M., & Greger, D. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252415 · Cited by 52 · see citing works
Working Out What Works: The Case of the Education Endowment Foundation in England
Edovald, T., & Nevill, C. (2020) · ECNU Review of Education · doi:10.1177/2096531120913039 · Cited by 49 · see citing works
The Effect-Size Benchmark That Matters Most: Education Interventions Often Fail
Kraft, M. A. (2023) · Educational Researcher · doi:10.3102/0013189x231155154 · Cited by 46 · see citing works
Wolf, B., & Harbatkin, E. (2023) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2022.2071364 · Cited by 45 · see citing works
How malleable are cognitive abilities? A critical perspective on popular brief interventions.
Moreau, D. (2021) · American Psychologist · doi:10.1037/amp0000872 · Cited by 44 · see citing works
Barone, C., Chambuleyron, E., Vonnak, R., & Assirelli, G. (2019) · Educational Research and Evaluation · doi:10.1080/13803611.2020.1814820 · Cited by 42 · see citing works
The dream of Sisyphus: Mathematics education in South Africa
Taylor, N. (2021) · South African Journal of Childhood Education · doi:10.4102/sajce.v11i1.911 · Cited by 42 · see citing works
A Commentary on Bowers (2020) and the Role of Phonics Instruction in Reading
Fletcher, J. M., Savage, R. S., & Vaughn, S. R. (2020) · Educational Psychology Review · doi:10.1007/s10648-020-09580-8 · Cited by 43 · see citing works
Algarni, B., & Lortie‐Forgues, H. (2022) · British Journal of Educational Technology · doi:10.1111/bjet.13250 · Cited by 52 · see citing works
Self-citationParra, J. D., & Edwards, D. B. (2024) · Critical Studies in Education · doi:10.1080/17508487.2024.2314118 · Cited by 36 · see citing works
When Do Growth Mindset Interventions Work?
Miller, D. I. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.08.005 · Cited by 36 · see citing works
From replication to substantiation: A complexity theory perspective
Al‐Hoorie, A. H., Hiver, P. V., Larsen‐Freeman, D., & Lowie, W. M. (2021) · Language Teaching · doi:10.1017/s0261444821000409 · Cited by 36 · see citing works
Promoting learning from null or negative results in prevention science trials
Axford, N., Berry, V., Lloyd, J., Hobbs, T., & Wyatt, K. (2022) · Prevention Science · doi:10.1007/s11121-020-01140-4 · Cited by 34 · see citing works
Evidence-based education: Objections and future directions
Dekker, I., & Meeter, M. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.941410 · Cited by 35 · see citing works
Theis, D., Sauerwein, M., & Fischer, N. (2019) · British Journal of Educational Psychology · doi:10.1111/bjep.12313 · Cited by 32 · see citing works
Experiment’s persistent failure in education inquiry, and why it keeps failing
Thomas, G. (2020) · British Educational Research Journal · doi:10.1002/berj.3660 · Cited by 35 · see citing works
Promoting a Growth Mindset in CS1: Does One Size Fit All? A Pilot Study
Quille, K., & Bergin, S. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387361 · Cited by 30 · see citing works
Impact of programming on primary mathematics learning
Laurent-Chevalier, M., Crisci, R., Bressoux, P., Chaachoua, H., Nurra, C., de Vries, E., & Tchounikine, P. (2022) · Learning and Instruction · doi:10.1016/j.learninstruc.2022.101667 · Cited by 30 · see citing works
The Impact of Summer Programs on Student Mathematics Achievement: A Meta-Analysis
Lynch, K., An, L., & Mancenido, Z. (2022) · Review of Educational Research · doi:10.3102/00346543221105543 · Cited by 34 · see citing works
Ryan, A., Prieto, E., Miller, A., & Gore, J. M. (2024) · Educational Research Review · doi:10.1016/j.edurev.2024.100620 · Cited by 34 · see citing works
Efficacy of a School-Based Comprehensive Intervention Program for Adolescents With Autism
Hume, K., Odom, S. L., Steinbrenner, J. R. D., DaWalt, L. S., Hall, L. J., Kraemer, B., Tomaszewski, B. R., Brum, C., Szidon, K., & Bolt, D. M. (2021) · Exceptional Children · doi:10.1177/00144029211062589 · Cited by 26 · see citing works
Miller, A., Gore, J. M., Wallington, C., Harris, J., Prieto, E., & Smith, M. (2019) · International Journal of Educational Research · doi:10.1016/j.ijer.2019.09.002 · Cited by 26 · see citing works
Isaacs, T., & Chalmers, H. (2023) · Language Teaching · doi:10.1017/s0261444823000411 · Cited by 26 · see citing works
Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education
Sims, S., Anders, J., Inglis, M., & Lortie‐Forgues, H. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2022.2090470 · Cited by 25 · see citing works
Self-citationA critical review of recent economics pedagogy literature, 2020–2021
Birdi, A., Cook, S., Elliott, C., Lait, A., Mehari, T., & Wood, M. (2023) · International Review of Economics Education · doi:10.1016/j.iree.2023.100264 · Cited by 25 · see citing works
On the misinterpretation of effect size
Simpson, A. P. (2019) · Educational Studies in Mathematics · doi:10.1007/s10649-019-09924-4 · Cited by 22 · see citing works
Learning, Marginalization, and Improving the Quality of Education in Low-income Countries
Online, L. A. T. B. O. T. P. (. 2., Wagner, D. A., Castillo, N. M., & Lewis, S. G. (2022) · Open Book Publishers · doi:10.11647/obp.0256 · Cited by 31 · see citing works
5. Reducing Inequality in Education Using “Smaller, Quicker, Cheaper” Assessments
Crouch, L. A., & Slade, T. S. (2022) · Open Book Publishers · doi:10.11647/obp.0256.05 · Cited by 20 · see citing works
Slattery, E. J., Butler, D., Marshall, K., Barrett, M. J., Hyland, N., O'Leary, M. R., & McAvinue, L. P. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.102003 · Cited by 32 · see citing works
Fisher, P. A., Barker, T. V., & Blaisdell, K. N. (2020) · Annual Review of Developmental Psychology · doi:10.1146/annurev-devpsych-121318-084958 · Cited by 20 · see citing works
A Path to Greater Credibility: Large-Scale Collaborative Education Research
Makel, M. C., Smith, K. N., McBee, M. T., Peters, S. J., & Miller, E. M. (2019) · AERA Open · doi:10.1177/2332858419891963 · Cited by 25 · see citing works
Education of Students with Disabilities, Science, and Randomized Controlled Trials
Odom, S. L. (2021) · Research and Practice for Persons with Severe Disabilities · doi:10.1177/15407969211032341 · Cited by 21 · see citing works
Groundhog day for inclusive education
Thomas, G., & Loxley, A. (2022) · Support for Learning · doi:10.1111/1467-9604.12406 · Cited by 19 · see citing works
Cahoon, A., Lira, C. J., Pérez, N. E., Pando, E. V. B., Campver, Y., García, D. S. P., & Simms, V. (2023) · Review of Educational Research · doi:10.3102/00346543231212491 · Cited by 19 · see citing works
Tipton, E., Spybrook, J. K., Fitzgerald, K. G., Wang, Q., & Davidson, C. (2020) · Educational Researcher · doi:10.3102/0013189x20960686 · Cited by 18 · see citing works
Dekker, I., Schippers, M. C., & van Schooten, E. (2021) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2023.2231440 · Cited by 25 · see citing works
AIED: From Cognitive Simulations to Learning Engineering, with Humans in the Middle
McNamara, D. S. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00349-y · Cited by 18 · see citing works
Ross, S. M., & Morrison, J. (2020) · ECNU Review of Education · doi:10.1177/2096531120916742 · Cited by 22 · see citing works
Cartwright, E. J., & Igudia, E. (2023) · Review of Development Economics · doi:10.1111/rode.13069 · Cited by 17 · see citing works
Hiver, P. V., & Nagle, C. (2024) · Annual Review of Applied Linguistics · doi:10.1017/s0267190524000060 · Cited by 16 · see citing works
Balancing Rigor, Replication, and Relevance: A Case for Multiple-Cohort, Longitudinal Experiments
Blazar, D., & Kraft, M. A. (2019) · AERA Open · doi:10.1177/2332858419876252 · Cited by 16 · see citing works
COVID derailed learning for 1.6 billion students. Here’s how schools can help them catch up
Pearson, H. (2022) · Nature · doi:10.1038/d41586-022-01387-7 · Cited by 16 · see citing works
The limits of inference: reassessing causality in international assessments
Rutkowski, D., Rutkowski, L. A., Thompson, G., & Canbolat, Y. (2024) · Large-scale Assessments in Education · doi:10.1186/s40536-024-00197-9 · Cited by 14 · see citing works
LURKING INFERENTIAL MONSTERS? QUANTIFYING SELECTION BIAS IN EVALUATIONS OF SCHOOL PROGRAMS
Weidmann, B., & Miratrix, L. W. (2020) · Journal of Policy Analysis and Management · doi:10.1002/pam.22236 · Cited by 13 · see citing works
Implementing Confidence Assessment in Low-Stakes, Formative Mathematics Assessments
Foster, C. F. (2021) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-021-10207-9 · Cited by 16 · see citing works
Compen, B., Witte, K. D., Declercq, K., & Schelfhout, W. (2022) · Education Economics · doi:10.1080/09645292.2022.2035322 · Cited by 14 · see citing works
Newbury, D. F., Mesa, C., Puglisi, M. L., Nash, M., Nag, S., Hulme, C., & Snowling, M. J. (2022) · Research Papers in Education · doi:10.1080/02671522.2022.2065526 · Cited by 12 · see citing works
Haag, P., Fantoni, T., & Dubal, S. (2022) · Journal of Intelligence · doi:10.3390/jintelligence10030064 · Cited by 12 · see citing works
The Returns to Education: A Meta‐Study
Clark, G., & Nielsen, C. A. A. (2026) · Kyklos · doi:10.1111/kykl.70041 · Cited by 12 · see citing works
Ozernov‐Palchik, O., Olson, H. A., Arechiga, X. M., Kentala, H., Solorio-Fielder, J. L., Wang, K. L., Torres, Y. C., Gardino, N. D., Dieffenbach, J. R., & Gabrieli, J. D. E. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.734375 · Cited by 12 · see citing works
What counts as ‘evidence’ in literacy education?
Derewianka, B., Harper, H., Parkin, B., Acevedo, C., Rose, D., Dare, B., Brisk, M. E., & Jones, P. T. (2024) · The Australian Journal of Language and Literacy · doi:10.1007/s44020-024-00071-9 · Cited by 11 · see citing works
Standards in education: reforms, stagnation and the need to rethink
Bolden, D. S., & Tymms, P. B. (2020) · Oxford Review of Education · doi:10.1080/03054985.2020.1781608 · Cited by 12 · see citing works
Realizing the promises of replication studies in education
Morrison, K. (2019) · Educational Research and Evaluation · doi:10.1080/13803611.2020.1838300 · Cited by 11 · see citing works
An Integrative Qualitative Framework: Improving Research Through Strategic Mapping
Coker, D. C. (2023) · International Research in Education · doi:10.5296/ire.v11i1.20921 · Cited by 11 · see citing works
Evaluating What Works in the Classroom: Best Practice and Future Opportunities
Bowen, A. E. J., Palmer, L. R. J., Perry, R. C., Thomas, M. S. C., Tolmie, A. K., Borst, G., & Herwegen, J. V. (2024) · Mind Brain and Education · doi:10.1111/mbe.12430 · Cited by 10 · see citing works
Missing, Presumed different: Quantifying the risk of Attrition Bias in Education Evaluations
Weidmann, B., & Miratrix, L. W. (2021) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12677 · Cited by 10 · see citing works
Launching Registered Report Replications in Computer Science Education Research
Brown, N. C. C., Marinus, E., & Cheuoua, A. H. (2022) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3501385.3543971 · Cited by 10 · see citing works
Learning from failure: A context‐informed perspective on RCTs
Coldwell, M., & Moore, N. (2024) · British Educational Research Journal · doi:10.1002/berj.3960 · Cited by 9 · see citing works
Stallasch, S. E., Lüdtke, O., Artelt, C., Hedges, L. V., & Brunner, M. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09898-7 · Cited by 9 · see citing works
Latent value in humiliation: A design thinking tool to enhance empathy in creative ideation
Svihla, V., & Kachelmeier, L. (2021) · International Journal of Design Creativity and Innovation · doi:10.1080/21650349.2021.1976677 · Cited by 9 · see citing works
Research–practice partnerships in education: Lessons from the United States
France, R. (2025) · British Educational Research Journal · doi:10.1002/berj.4132 · Cited by 8 · see citing works
Replanteando la acción social por la música
Baker, G. (2022) · Open Book Publishers · doi:10.11647/obp.0263 · Cited by 8 · see citing works
Multiverses, Myths, and the Big Bang
Coker, D. C. (2022) · Advances in educational marketing, administration, and leadership book series · doi:10.4018/978-1-6684-4144-2.ch022 · Cited by 8 · see citing works
Evidence, Policy, Education, and Neuroscience—The State of Play in the UK
Thomas, M. S. C., Howard‐Jones, P. A., Dudman‐Jones, J., Palmer, L. R. J., Bowen, A. E. J., & Perry, R. C. (2024) · Mind Brain and Education · doi:10.1111/mbe.12423 · Cited by 9 · see citing works
Luyten, H., Merrell, C. H., & Tymms, P. B. (2020) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2020.100939 · Cited by 8 · see citing works
Improving SEL outcomes for US 9th graders: Results from a District Wide School Reform
Rosen, J. A., Hong, Y., Rutledge, S. A., Cannata, M. A., Rotermund, S., & Leu, K. (2022) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2022.101212 · Cited by 7 · see citing works
Focusing on spontaneous focusing
Inglis, M. (2020) · Mathematical Thinking and Learning · doi:10.1080/10986065.2020.1818472 · Cited by 8 · see citing works
Self-citationOpen Science 2.0: Large-Scale Collaborative Education ResearchPreprint
Makel, M. C., Smith, K. N., McBee, M. T., Peters, S. J., & Miller, E. M. (2019) · doi:10.31234/osf.io/ypmjg · Cited by 9 · see citing works
Performance-based accountability systems at the organizational level: effects of a school program
Andersen, S. C., Hvidman, U., Junge, S. Y., & Rangvid, B. S. (2025) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muaf007 · Cited by 7 · see citing works
3. Teaching at the Bottom of the Pyramid
Akyeampong, K. (2022) · Open Book Publishers · doi:10.11647/obp.0256.03 · Cited by 7 · see citing works
4. Improving the Impact of Educational Technologies on Learning Within Low-Income Contexts
Castillo, N. M., Adam, T., & Haßler, B. (2022) · Open Book Publishers · doi:10.11647/obp.0256.04 · Cited by 7 · see citing works
A Recipe for Disappointment: Policy, Effect Size, and the Winner’s Curse
Simpson, A. P. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2022.2066588 · Cited by 7 · see citing works
Connecting learning environments to learning: Two examples from children’s mathematics
Siegler, R. S. (2024) · Developmental Review · doi:10.1016/j.dr.2024.101138 · Cited by 7 · see citing works
Sims, S., Anders, J., & Zieger, L. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2022.2051652 · Cited by 7 · see citing works
Petty, C. S., Eddy, C. M., & Pratt, S. S. (2022) · Learning Environments Research · doi:10.1007/s10984-022-09421-1 · Cited by 7 · see citing works
Kanefke, J., & Schukajlow, S. (2023) · Journal of Mathematics Teacher Education · doi:10.1007/s10857-023-09581-8 · Cited by 8 · see citing works
Sabatini, J., Park, J. H., Chakraborty, A., Graesser, A. C., Macintyre, T., Chandrasekharan, M., & Ogunniran, M. O. (2022) · UNESCO MGIEP eBooks · doi:10.56383/sdro8173 · Cited by 6 · see citing works
Jasińska, K. K., & Guei, S. (2022) · Open Book Publishers · doi:10.11647/obp.0256.13 · Cited by 6 · see citing works
How can we optimise learning from trials in child and adolescent mental health?
Axford, N., Berry, V., Lloyd, J., & Wyatt, K. M. (2022) · Evidence-Based Mental Health · doi:10.1136/ebmental-2022-300500 · Cited by 6 · see citing works
Barone, C., Fougère, D., & Martel, K. (2024) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2023.2283475 · Cited by 5 · see citing works
Exploring randomised controlled trials in education: key features and selected examples from Ireland
Slattery, E. J., Clerkin, A., Gilleece, L., Butler, D., Marshall, K., & Hedges, L. V. (2025) · Irish Educational Studies · doi:10.1080/03323315.2025.2523301 · Cited by 6 · see citing works
- Authors
- Lortie-Forgues, H., & Inglis, M.
- Affiliations
- University of York, UK
- Loughborough University (GB)
- ORCID
- Inglis, Matthew
- Venue
- Educational Researcher
- Published
- 2019-03-11
- Volume
- 48
- Issue
- 3
- Pages
- 158-166
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2919428740
- Cited by
- 234
- References
- 44
- Citations by year
- 2026: 19 · 2025: 20 · 2024: 27 · 2023: 19 · 2022: 64 · 2021: 39 · 2020: 30 · 2019: 16
- FWCI
- 24.93
- Type
- journal article
- Language
- en
- Open access
- green
- orcid.org
- Locations
- Educational Researcher (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Loughborough University Institutional Repository (Loughborough University) (submittedVersion, cc-by-nc-nd)
- Loughborough University Research Repository (Loughborough University) (submittedVersion, cc-by-nc-nd)
- Funders
- university of york
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › School Choice and Performance
- Social Sciences › Education › Educational Research and Methods
- Keywords
- educational intervention, effect size, confidence intervals, Bayes factor, statistical power, Education Endowment Foundation, evidence-based education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
