CoChat

A Framework for Learning From Null Results

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.

Abstract

A substantial number of randomized trials of educational interventions that have been conducted over the past two decades have produced null results, with either no impact or an unreliable estimate of impact on student achievement or other outcomes of interest. The investment of time and money spent implementing such trials warrants more useful information than simply “this didn’t work.” In this article, we propose a framework for defining null results and interpreting them and then propose a method for systematically examining a set of potential reasons for a study’s findings. The article builds on prior work on the topic and synthesizes it into a common framework designed to help the field improve both the design and interpretation of randomized trials.

Overview

Integrity signals

Registry

Crossref · 10.3102/0013189x19891955

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 3

  1. Aberson C., 2002, Journal of Articles in Support of the Null Hypothesis, 1, 36
  2. Cook T., 2006, What works in publicly funded prekindergarten education?
  3. 10.2466/pr0.1997.80.1.337 · DOI

Cited by

See all 70

Metadata

Authors
Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A.
Affiliations
  • University of Michigan (US)
  • Manpower Demonstration Research Corporation (US)
  • New York University (US)
Corresponding author
Jacob, Robin T.
Venue
Educational Researcher
Published
2019-12
Volume
48
Issue
9
Pages
580-589
ISSN
  • 0013-189X (ISSN-L)
  • 1935-102X
OpenAlex
W2995058077
Cited by
65
References
48
Citations by year
2026: 8 · 2025: 12 · 2024: 8 · 2023: 12 · 2022: 9 · 2021: 8 · 2020: 5 · 2019: 3
FWCI
16.86
Type
journal article
Language
en
Open access
closed
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
Educational Researcher (publishedVersion)
Topics
  • Social Sciences › Education › Educational Research and Methods
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
Keywords
negative results, randomized controlled trials, educational intervention, academic performance, trial interpretation, trial design, effect size estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.