A Framework for Learning From Null Results
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
A substantial number of randomized trials of educational interventions that have been conducted over the past two decades have produced null results, with either no impact or an unreliable estimate of impact on student achievement or other outcomes of interest. The investment of time and money spent implementing such trials warrants more useful information than simply “this didn’t work.” In this article, we propose a framework for defining null results and interpreting them and then propose a method for systematically examining a set of potential reasons for a study’s findings. The article builds on prior work on the topic and synthesizes it into a common framework designed to help the field improve both the design and interpretation of randomized trials.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The failure of Restorative Practices to improve overall behavioral and school climate outcomes in its first year of implementation at JCPS may have been partially due to lack of treatment contrast between the treatment and comparison schools because the comparison condition provided similar services to students (Jacob et al., 2019).”
From Restorative Practices: Using local evidence on costs and student outcomes to inform school district decisions about behavioral interventions (2022) · cites this paper as Jacob et al., 2019
In the citing paper
“This lack of information leads to a lack of clarity, for example concerning the comparator and the treatment–control contrast [50, 51].”
Citations
Cites
(2002) · Cited by 13,211
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“This is both because studies of interventions that do not work are less likely to get published and program developers are more likely to try to publish evaluations that show that their program is effective (Franco et al., 2014; Rosenthal, 1979).”
BackgroundThe ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 1×
“DECEmbER 2019 583 that an exclusive focus on the statistical significance of effects without considering the substantive magnitude of those effects is misplaced (e.g., Wasserstein & Lazar, 2016).”
BackgroundEquivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 3,001
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · cited 2×
“A reading intervention that has positive impacts on decoding skills but not reading comprehension skills is effective at improving decoding skills but not an effective reading intervention overall; an intervention that has no main impacts but does have impacts for the boys in the sample is an effective intervention fo…”
BackgroundLipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 1×
“They found that studies that were funded earliest (between 2002 and 2004) had minimal detectable effect sizes between 0.40 and 0.90, which means that they likely were not sufficiently powered to detect meaningful effects (Lipsey & Wilson, 1993; Schochet, 2008a; Spybrook & Raudenbush, 2009).”
BackgroundInfluentialPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · cited 1×
“This is both because studies of interventions that do not work are less likely to get published and program developers are more likely to try to publish evaluations that show that their program is effective (Franco et al., 2014; Rosenthal, 1979).”
BackgroundContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · cited 2×
“In medicine, for example, 50% to 80% of Stage II trials that find positive results are not able to replicate these results in larger randomized trials (Chan et al., 2008; Ioannidis, 2005; Zia et al., 2005).”
BackgroundExperimental Estimates of Education Production Functions
Krueger, A. B. (1999) · The Quarterly Journal of Economics · doi:10.1162/003355399556052 · Cited by 1,120
Getting to Scale with Good Educational Practice
Elmore, R. (1996) · Harvard Educational Review · doi:10.17763/haer.66.1.g73266758j348t33 · Cited by 1,058 · cited 4×
“Elmore (1996) similarly argued that it is school organizational structures and incentive systems that make it difficult to change the core practices of teaching. This same sentiment was echoed by D. Olson (2004), who suggested that a closer examination of “the school’s place in the institutional structures of a bureau…”
BackgroundInfluentialMind-Set Interventions Are a Scalable Treatment for Academic Underachievement
Paunesku, D., Walton, G. M., Romero, C., Smith, E. N., Yeager, D. S., & Dweck, C. S. (2015) · Psychological Science · doi:10.1177/0956797615571017 · Cited by 981 · cited 1×
“Mindset training targets students’ core beliefs about school and learning and has been shown to have a small positive impact on students’ GPA and successful course completion (Paunesku et al., 2015).”
BackgroundLipsey, M. W. (2009) · Victims & Offenders · doi:10.1080/15564880802612573 · Cited by 962 · cited 2×
“90, which means that they likely were not sufficiently powered to detect meaningful effects (Lipsey & Wilson, 1993; Schochet, 2008a; Spybrook & Raudenbush, 2009). The precision of the studies improved over time such that studies funded between 2006 and 2008 had minimum detectable effects ranging from about 0.18 to 0.4…”
BackgroundInfluentialEmpirical Benchmarks for Interpreting Effect Sizes in Research
Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008) · Child Development Perspectives · doi:10.1111/j.1750-8606.2008.00061.x · Cited by 950 · cited 5×
“…with highly aligned measures more likely to yield larger impact estimates than those that are less closely aligned and proximal measures more likely to yield positive effects than more distal measures (e.g., Hill et al., 2008; Marzano, 2014; R. B. Olson et al., 2011; Slavin & Madden, 2011).”
BackgroundInfluentialRaver, C. C., Jones, S. M., Li-Grining, C., Zhai, F., Bub, K., & Pressler, E. (2011) · Child Development · doi:10.1111/j.1467-8624.2010.01561.x · Cited by 754 · cited 1×
“The most effective school-based interventions designed to impact executive function have only had impacts on measures of executive function equal to around half a standard deviation (e.g., Raver et al., 2011).”
BackgroundFacts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · cited 1×
“Replication is important (Makel & Plucker, 2014), and in addition to thinking more carefully about the findings of individual studies, broader conclusions about an intervention’s effectiveness should be made by systematically reviewing and integrating findings based on multiple studies.”
BackgroundJacob, R., & Parkinson, J. (2015) · Review of Educational Research · doi:10.3102/0034654314561338 · Cited by 538 · cited 1×
“…mean that at most, such interventions might be expected to have impacts on achievement of no more than 0.01 to 0.06 SD. Likewise, a meta-analysis by Jacob and Parkinson (2015) found that after controlling for background characteristics and IQ, the largest association between executive function and…”
BackgroundSelf-citationThe Relationship Between Sample Sizes and Effect Sizes in Systematic Reviews in Education
Slavin, R., & Smith, D. (2009) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373709352369 · Cited by 360 · cited 2×
“However, as has been shown in education and other fields, sample size has a strong negative correlation with effect size (Slavin & Smith, 2009).”
BackgroundEffective programs for struggling readers: A best-evidence synthesis
Slavin, R. E., Lake, C., Davis, S., & Madden, N. A. (2011) · Educational Research Review · doi:10.1016/j.edurev.2010.07.002 · Cited by 353 · cited 2×
“Meta-analysis has shown there is often substantial variability in the observed effects across multiple studies of the same or very similar interventions (e.g., Lipsey, 2009; Slavin et al., 2011).”
BackgroundInfluentialProfessional Development for Science Teachers
Wilson, S. M. (2013) · Science · doi:10.1126/science.1230725 · Cited by 325 · cited 1×
“And Wilson (2013) noted that although existing research on effective professional development identified factors that led to effective professional development, the complexity of the U.S. educational system often thwarts efforts to support teachers, and as a result, interventions that take a more…”
BackgroundStatistical power for random assignment evaluations of education programs
Schochet, P. Z. (2008) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998607302714 · Cited by 216 · cited 3×
“They found that studies that were funded earliest (between 2002 and 2004) had minimal detectable effect sizes between 0.40 and 0.90, which means that they likely were not sufficiently powered to detect meaningful effects (Lipsey & Wilson, 1993; Schochet, 2008a; Spybrook & Raudenbush, 2009).”
BackgroundInfluentialGersten, R., Dimino, J., Jayanthi, M., Kim, J. S., & Santoro, L. E. (2010) · American Educational Research Journal · doi:10.3102/0002831209361208 · Cited by 198 · cited 1×
“Most studies of professional development programs found relatively modest impacts on instructional practice and teacher knowledge, in the range of 0.10 to 0.60 SD (e.g., Garet et al., 2011; Gersten et al., 2010; Jacob et al., 2017; Schneider & Meyer, 2012).”
BackgroundInfluentialGaret, M. S., Wayne, A. J., Stancavage, F., Taylor, J., Eaton, M., Walters, K., Song, M., Brown, S., Hurlburt, S., Zhu, P., et al. (2011) · National Center for Education Evaluation and Regional Assistance · Cited by 160
Self-citationZia, M. I., Siu, L. L., Pond, G. R., & Chen, E. X. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.06.679 · Cited by 157 · cited 1×
“In medicine, for example, 50% to 80% of Stage II trials that find positive results are not able to replicate these results in larger randomized trials (Chan et al., 2008; Ioannidis, 2005; Zia et al., 2005).”
BackgroundJacob, R., Hill, H., & Corey, D. (2017) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2016.1273411 · Cited by 154 · cited 1×
“Most studies of professional development programs found relatively modest impacts on instructional practice and teacher knowledge, in the range of 0.10 to 0.60 SD (e.g., Garet et al., 2011; Gersten et al., 2010; Jacob et al., 2017; Schneider & Meyer, 2012).”
BackgroundSelf-citationMeasures Inherent to Treatments in Program Effectiveness Reviews
Slavin, R., & Madden, N. A. (2011) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.558986 · Cited by 124 · cited 2×
“Slavin and Madden (2011) showed that among WWC studies of beginning reading programs, the outcome measures that were closely aligned with the intervention yielded weighted effect sizes of 0.51 SD, whereas the more general outcome measures had average effect sizes equal to 0.06.”
BackgroundInfluentialEvidence-Based Practices in a Changing World: Reconsidering the Counterfactual in Education Research
Lemons, C. J., Fuchs, D., Gilbert, J. K., & Fuchs, L. S. (2014) · Educational Researcher · doi:10.3102/0013189x14539189 · Cited by 121 · cited 2×
“Researchers have approached the problem of null results from a variety of perspectives over the past decade, noting a number of potential contributors (see e.g., Landis et al., 2014; Lemons et al., 2014; Shager et al., 2013).”
BackgroundBloom, H. S., & Weiland, C. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2594430 · Cited by 115 · cited 2×
“For example, recent work by Bloom and Weiland (2015) showed that past estimates of the effectiveness of Head Start programs, which focused only on overall average impacts, masked a wide range of relative program effectiveness for specific subgroups of students, particularly for English Language Learners and those begi…”
ResultThe Triumph of Hope Over Experience in the Search for “What Works”: A Response to Slavin
Olson, D. R. (2004) · Educational Researcher · doi:10.3102/0013189x033001024 · Cited by 114 · cited 2×
“In a similar vein, D. Olson (2004) suggested that the counterfactual is likely to both change over time or change in response to the treatment, making it difficult to identify the factors that “uniquely define the treatment” itself.”
BackgroundMethodBarnett, W. S., & Hustedt, J. T. (2005) · Infants & Young Children · doi:10.1097/00001163-200501000-00003 · Cited by 105 · cited 1×
“…of high-quality preschool, where long-term follow-up studies have revealed impacts on a variety of outcomes, including a reduction in grade retention, special education placements, delinquency, and incarceration rates many years after the completion of the program (e.g., Barnett & Hustedt, 2005).”
BackgroundTechnical Methods Report: Guidelines for Multiple Testing in Impact Evaluations. NCEE 2008-4018.
Schochet, P. Z. (2008) · National Center for Education Evaluation and Regional Assistance · Cited by 97
Spybrook, J., & Raudenbush, S. W. (2009) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373709339524 · Cited by 88 · cited 3×
“They found that studies that were funded earliest (between 2002 and 2004) had minimal detectable effect sizes between 0.40 and 0.90, which means that they likely were not sufficiently powered to detect meaningful effects (Lipsey & Wilson, 1993; Schochet, 2008a; Spybrook & Raudenbush, 2009).”
BackgroundInfluentialInterpreting Null Results: Improving Presentation and Conclusions with Confidence Intervals 1
Aberson, C. L. (2002) · Cited by 87
PUBLICATION BIAS AGAINST NULL RESULTS
Hubbard, R., & Armstrong, J. S. (1997) · Psychological Reports · doi:10.2466/pr0.80.1.337-338 · Cited by 70 · cited 1×
“More generally, confidence intervals are a useful tool for interpreting findings because they contain information not only about the precision of the estimate but also about the direction and magnitude of the impacts (Aberson, 2002; Hubbard & Armstrong, 1997).”
MethodShager, H. M., Schindler, H. S., Magnuson, K. A., Duncan, G. J., Yoshikawa, H., & Hart, C. M. D. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712462453 · Cited by 65 · cited 1×
“Researchers have approached the problem of null results from a variety of perspectives over the past decade, noting a number of potential contributors (see e.g., Landis et al., 2014; Lemons et al., 2014; Shager et al., 2013).”
BackgroundLandis, R. S., James, L. R., Lance, C. E., Pierce, C. A., & Rogelberg, S. G. (2014) · Journal of Business and Psychology · doi:10.1007/s10869-014-9347-8 · Cited by 63 · cited 2×
“Lee et al. (2013), in their study of the impact of No Child Left Behind school interventions in New York, noted that overlooking contextual factors, such as the social and racial composition of the school or a district’s internal capacity for change, can prove “fatal” in the implementation of any reform effort.”
BackgroundChan, J. K., Ueda, S. M., Sugiyama, V. E., Stave, C. D., Shin, J. Y., Monk, B. J., Sikic, B. I., Osann, K., & Kapp, D. S. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2007.14.8874 · Cited by 62 · cited 4×
“In medicine, for example, 50% to 80% of Stage II trials that find positive results are not able to replicate these results in larger randomized trials (Chan et al., 2008; Ioannidis, 2005; Zia et al., 2005).”
BackgroundInfluentialGrade Assignments and the Teacher Pipeline: A Low-Cost Lever to Improve Student Achievement?
Blazar, D. (2015) · Educational Researcher · doi:10.3102/0013189x15580944 · Cited by 61 · cited 3×
“An article by Blazar (2015) suggested that a 1 SD increase in teachers’ ambitious mathematics practice is associated with around a 0.”
BackgroundInfluentialReading First Impact Study: Interim Report
Gamse, B., Bloom, H. S., Kemple, J. J., Jacob, R. T., Boulay, B. A., Bozzi, L., Caswell, L., Horst, M. A., Smith, W. C., St, R. G., et al. (2008) · Issue Lab (Candid) · Cited by 51
Self-citationLeveraging Volunteers: An Experimental Evaluation of a Tutoring Program for Struggling Readers
Jacob, R., Armstrong, C., Bowden, A. B., & Pan, Y. (2016) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2016.1138560 · Cited by 38 · cited 1×
“This resulted in a situation in which 65% of the control group was also receiving some type of supplemental reading instruction, likely diluting the estimated impact of the intervention (Jacob et al., 2016).”
BackgroundSelf-citationSpybrook, J. K. (2013) · The Journal of Experimental Education · doi:10.1080/00220973.2013.813364 · Cited by 32 · cited 1×
“Spybrook (2014) also showed that studies typically do not have enough precision to detect differences in intervention effects across key subgroups of interest, such as rural and urban schools or high- and low-poverty districts, meaning that it will be difficult to identify if programs are effective…”
BackgroundSchochet, P. Z., Puma, M., & Deke, J. (2014) · National Center for Education Evaluation and Regional Assistance · Cited by 32
Schneider, M. C., & Meyer, J. P. (2012) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v8i17.291 · Cited by 26 · cited 1×
“Most studies of professional development programs found relatively modest impacts on instructional practice and teacher knowledge, in the range of 0.10 to 0.60 SD (e.g., Garet et al., 2011; Gersten et al., 2010; Jacob et al., 2017; Schneider & Meyer, 2012).”
BackgroundInfluentialEvaluating the impact of NCLB school interventions in New York State: Does one size fit all?
Lee, J., Shin, H., & Amo, L. (2013) · Education Policy Analysis Archives · doi:10.14507/epaa.v21n67.2013 · Cited by 19 · cited 1×
“Lee et al. (2013), in their study of the impact of No Child Left Behind school interventions in New York, noted that overlooking contextual factors, such as the social and racial composition of the school or a district’s internal capacity for change, can prove “fatal” in the implementation of any…”
BackgroundThe Late Pretest Problem in Randomized Control Trials of Education Interventions
Schochet, P. Z. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609353112 · Cited by 10 · cited 1×
“…can also impact the degree to which impacts can be detected; if pretest data are collected after the start of the intervention or posttests are administered before the intervention has concluded, estimates of program impact may be smaller than they otherwise might have been (e.g., Schochet, 2010).”
BackgroundWest, M. R., Morton, B. A., & Herlihy, C. M. (2016) · Grantee Submission · Cited by 10
Proximal Versus Distal Validity Coefficients for Teacher Observational Instruments
Marzano, R. J. (2014) · The Teacher Educator · doi:10.1080/08878730.2014.885783 · Cited by 8 · cited 2×
“Finally, the degree of alignment between the treatment and the outcome measure will influence the size of the impact estimate, with highly aligned measures more likely to yield larger impact estimates than those that are less closely aligned and proximal measures more likely to yield positive effects than more distal…”
BackgroundEstimating the Impacts of Educational Interventions Using State Tests or Study-Administered Tests
Olsen, R. B., Unlu, F., Price, C. S., Jaciw, A. P., & Bachman, M. (2011) · PsycEXTRA Dataset · doi:10.1037/e690452011-001 · Cited by 4
Glewwe, P. W., & West, K. L. (2013) · AEA Randomized Controlled Trials · doi:10.1257/rct.167-4.0 · Cited by 1
Glewwe, P. W., & West, K. (2013) · AEA Randomized Controlled Trials · doi:10.1257/rct.167 · Cited by 1
Research works printed but not linked: 3
- Aberson C., 2002, Journal of Articles in Support of the Null Hypothesis, 1, 36
- Cook T., 2006, What works in publicly funded prekindergarten education?
- 10.2466/pr0.1997.80.1.337 · DOI
Cited by
Replication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 126
“…success have been proposed, each with its strengths and limitations (Schauer & Hedges, 2021), including interpreting confidence intervals (Jacob et al., 2019; Zwaan et al., 2018), a combination of sample and effect sizes (Simonsohn, 2015), and a replication Bayes factor (Zwaan et al.,…”
BackgroundUsing Implementation Fidelity to Aid in Interpreting Program Impacts: A Brief Review
Hill, H., & Erickson, A. (2019) · Educational Researcher · doi:10.3102/0013189x19891436 · Cited by 88
On Partial Versus Full Mediation and the Importance of Effect Sizes
Ledermann, T., Rudaz, M., & Fritz, M. S. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251355585 · Cited by 51
“There may also be unmeasured subpopulations for which the null result is not true (see also Jacob et al., 2019; Landis et al., 2014).”
Ming, N. C., & Goldenberg, L. B. (2021) · Review of Research in Education · doi:10.3102/0091732x21990620 · Cited by 56
“Some propose different thresholds (e.g., Gelman & Robert, 2014; Ioannidis, 2019; Johnson, 2013; Mayo, 2019), while others object to creating arbitrary thresholds (Ziliak & McCloskey, 2008) or advocate learning from null results (e.g., Jacob et al., 2019).”
BackgroundZucker, T. A., Jacbos, E., & Cabell, S. Q. (2021) · Early Education and Development · doi:10.1080/10409289.2020.1839288 · Cited by 46
“Education researchers often argue that a cause of null findings is not that an intervention’s theory of change is entirely flawed but that change is only possible under ideal conditions (e.g., highly skilled teachers, schools with prerequisite “readiness” factors in place; Jacob et al., 2019).”
Background
Metadata
- Authors
- Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A.
- Affiliations
- University of Michigan (US)
- Manpower Demonstration Research Corporation (US)
- New York University (US)
- Corresponding author
- Jacob, Robin T.
- Venue
- Educational Researcher
- Published
- 2019-12
- Volume
- 48
- Issue
- 9
- Pages
- 580-589
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2995058077
- Cited by
- 65
- References
- 48
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 8 · 2023: 12 · 2022: 9 · 2021: 8 · 2020: 5 · 2019: 3
- FWCI
- 16.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational Researcher (publishedVersion)
- Topics
- Social Sciences › Education › Educational Research and Methods
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- negative results, randomized controlled trials, educational intervention, academic performance, trial interpretation, trial design, effect size estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.