A Framework for Learning From Null Results
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
A substantial number of randomized trials of educational interventions that have been conducted over the past two decades have produced null results, with either no impact or an unreliable estimate of impact on student achievement or other outcomes of interest. The investment of time and money spent implementing such trials warrants more useful information than simply “this didn’t work.” In this article, we propose a framework for defining null results and interpreting them and then propose a method for systematically examining a set of potential reasons for a study’s findings. The article builds on prior work on the topic and synthesizes it into a common framework designed to help the field improve both the design and interpretation of randomized trials.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The failure of Restorative Practices to improve overall behavioral and school climate outcomes in its first year of implementation at JCPS may have been partially due to lack of treatment contrast between the treatment and comparison schools because the comparison condition provided similar services to students (Jacob et al., 2019).”
From Restorative Practices: Using local evidence on costs and student outcomes to inform school district decisions about behavioral interventions (2022) · cites this paper as Jacob et al., 2019
In the citing paper
“This lack of information leads to a lack of clarity, for example concerning the comparator and the treatment–control contrast [50, 51].”
Citations
Cites
(2002) · Cited by 13,211
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“This is both because studies of interventions that do not work are less likely to get published and program developers are more likely to try to publish evaluations that show that their program is effective (Franco et al., 2014; Rosenthal, 1979).”
BackgroundThe ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 1×
“DECEmbER 2019 583 that an exclusive focus on the statistical significance of effects without considering the substantive magnitude of those effects is misplaced (e.g., Wasserstein & Lazar, 2016).”
BackgroundEquivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 3,001
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · cited 2×
“A reading intervention that has positive impacts on decoding skills but not reading comprehension skills is effective at improving decoding skills but not an effective reading intervention overall; an intervention that has no main impacts but does have impacts for the boys in the sample is an effective intervention fo…”
BackgroundLipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 1×
“They found that studies that were funded earliest (between 2002 and 2004) had minimal detectable effect sizes between 0.40 and 0.90, which means that they likely were not sufficiently powered to detect meaningful effects (Lipsey & Wilson, 1993; Schochet, 2008a; Spybrook & Raudenbush, 2009).”
BackgroundInfluentialPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · cited 1×
“This is both because studies of interventions that do not work are less likely to get published and program developers are more likely to try to publish evaluations that show that their program is effective (Franco et al., 2014; Rosenthal, 1979).”
BackgroundContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · cited 2×
“In medicine, for example, 50% to 80% of Stage II trials that find positive results are not able to replicate these results in larger randomized trials (Chan et al., 2008; Ioannidis, 2005; Zia et al., 2005).”
BackgroundExperimental Estimates of Education Production Functions
Krueger, A. B. (1999) · The Quarterly Journal of Economics · doi:10.1162/003355399556052 · Cited by 1,120
Getting to Scale with Good Educational Practice
Elmore, R. (1996) · Harvard Educational Review · doi:10.17763/haer.66.1.g73266758j348t33 · Cited by 1,058 · cited 4×
“Elmore (1996) similarly argued that it is school organizational structures and incentive systems that make it difficult to change the core practices of teaching. This same sentiment was echoed by D. Olson (2004), who suggested that a closer examination of “the school’s place in the institutional structures of a bureau…”
BackgroundInfluential
Research works printed but not linked: 3
- Aberson C., 2002, Journal of Articles in Support of the Null Hypothesis, 1, 36
- Cook T., 2006, What works in publicly funded prekindergarten education?
- 10.2466/pr0.1997.80.1.337 · DOI
Cited by
Replication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 126
“…success have been proposed, each with its strengths and limitations (Schauer & Hedges, 2021), including interpreting confidence intervals (Jacob et al., 2019; Zwaan et al., 2018), a combination of sample and effect sizes (Simonsohn, 2015), and a replication Bayes factor (Zwaan et al.,…”
BackgroundUsing Implementation Fidelity to Aid in Interpreting Program Impacts: A Brief Review
Hill, H., & Erickson, A. (2019) · Educational Researcher · doi:10.3102/0013189x19891436 · Cited by 88
On Partial Versus Full Mediation and the Importance of Effect Sizes
Ledermann, T., Rudaz, M., & Fritz, M. S. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251355585 · Cited by 51
“There may also be unmeasured subpopulations for which the null result is not true (see also Jacob et al., 2019; Landis et al., 2014).”
Ming, N. C., & Goldenberg, L. B. (2021) · Review of Research in Education · doi:10.3102/0091732x21990620 · Cited by 55
“Some propose different thresholds (e.g., Gelman & Robert, 2014; Ioannidis, 2019; Johnson, 2013; Mayo, 2019), while others object to creating arbitrary thresholds (Ziliak & McCloskey, 2008) or advocate learning from null results (e.g., Jacob et al., 2019).”
BackgroundZucker, T. A., Jacbos, E., & Cabell, S. Q. (2021) · Early Education and Development · doi:10.1080/10409289.2020.1839288 · Cited by 46
“Education researchers often argue that a cause of null findings is not that an intervention’s theory of change is entirely flawed but that change is only possible under ideal conditions (e.g., highly skilled teachers, schools with prerequisite “readiness” factors in place; Jacob et al., 2019).”
Background
Metadata
- Authors
- Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A.
- Affiliations
- University of Michigan (US)
- Manpower Demonstration Research Corporation (US)
- New York University (US)
- Corresponding author
- Jacob, Robin T.
- Venue
- Educational Researcher
- Published
- 2019-12
- Volume
- 48
- Issue
- 9
- Pages
- 580-589
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2995058077
- Cited by
- 65
- References
- 48
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 8 · 2023: 12 · 2022: 9 · 2021: 8 · 2020: 5 · 2019: 3
- FWCI
- 16.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational Researcher (publishedVersion)
- Topics
- Social Sciences › Education › Educational Research and Methods
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- negative results, randomized controlled trials, educational intervention, academic performance, trial interpretation, trial design, effect size estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Framework for Learning From Null Results. https://app.cochat.ai/papers/10.3102/0013189x19891955
- The paper
- Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A. (2019). A Framework for Learning From Null Results. Educational Researcher. https://doi.org/10.3102/0013189x19891955
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.