CoChat
Sign Up
Cited by 208Online Learning and AnalyticsOnline and Blended LearningHigher Education Research Studies

The DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index, and 1 are not checked yet.

Scaling up behavioral science interventions in online education

Open access
Kizilcec, René F., Reich, Justin, Yeomans, Michael, Dann, Christoph, Brunskill, Emma, Lopez, Glenn, Turkay, Selen, Williams, Joseph Jay, Tingley, Dustin·2020·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1921417117PMC PMC7334459
Open this paper in CoChat — free accountOpen PDF

Abstract

Significance: Low persistence in educational programs is a major obstacle to social mobility. Scientists have proposed many scalable interventions to support students learning online. In one of the largest international field experiments in education, we iteratively tested established behavioral science interventions and found small benefits depending on individual and contextual characteristics. Forecasting intervention efficacy using state-of-the-art methods yields limited improvements. Online education provides unprecedented access to learning opportunities, as evidenced by its role during the 2020 coronavirus pandemic, but adequately supporting diverse students will require more than a light-touch intervention. Our findings encourage funding agencies and researchers conducting large-scale field trials to consider dynamic investigations to uncover and design for contextual heterogeneity to complement static investigations of overall effects.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1921417117

Cites

35 of 36 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Average student activity (count of course platform events) in the first 3 wk after exposure to each intervention. Points show covariate-adjusted means on a logarithmic scale (to match the log-transformed outcome in the regression model) with cluster-robust SE bars.
Fig. 1. Average student activity (count of course platform events) in the first 3 wk after exposure to each intervention. Points show covariate-adjusted means on a logarithmic scale (to match the log-transformed outcome in the regression model) with cluster-robust SE bars.CC BY-NC-ND · PubMed Central
Fig. 2. Average course completion rate in all waves in the value-relevance intervention and control condition by student context (more vs. less developed country) and course context (with vs. without global achievement gap). Bars show covariate-adjusted means with cluster-robust SE bars.
Fig. 2. Average course completion rate in all waves in the value-relevance intervention and control condition by student context (more vs. less developed country) and course context (with vs. without global achievement gap). Bars show covariate-adjusted means with cluster-robust SE bars.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. C. R. Sunstein, R. H. Thaler, Nudge: Improving Deisions About Health, Wealth, and Happiness, (Yale Univeresity Press, New Haven, 2008).
  2. C. Zhenghao ., Who’s benefiting from MOOCs, and why. Harv. Bus. Rev. 25, 2–8 (2015).
  3. 10.1353/bsp.2015.0011 · DOI

Cited by

Sorted by citations

See all 208

Related

See all

Metadata

Authors
Kizilcec, R. F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., Turkay, S., Williams, J. J., & Tingley, D.
Affiliations
  • Cornell University (US)
  • Massachusetts Institute of Technology (US)
  • Harvard University (US)
  • Carnegie Mellon University (US)
  • Stanford University (US)
  • School of Computer Science, Queensland University of Technology, Brisbane City, QLD 4000, Australia;
  • University of Toronto (CA)
Corresponding author
  • Kizilcec, René F.
  • Reich, Justin
  • Yeomans, Michael
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2020-06-15
Volume
117
Issue
26
Pages
14900-14905
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1921417117
PMC
PMC7334459
OpenAlex
W3034223452
Cited by
208
References
35
Citations by year
2026: 23 · 2025: 30 · 2024: 28 · 2023: 30 · 2022: 39 · 2021: 49 · 2020: 8
FWCI
28.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • QUT ePrints (Queensland University of Technology) (submittedVersion, cc-by-nc-nd)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion, cc-by-nc-nd)
Funders
  • National Science Foundation (1646978)
  • Stanford University (n/a)
  • Microsoft (n/a)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › Education › Online and Blended Learning
  • Social Sciences › Education › Higher Education Research Studies
Keywords
online learning, student persistence, course completion, self-regulation interventions, scaling up interventions, preregistration, MOOCs
MeSH terms
Behavior, Behavioral Sciences, Goals, Humans, Research, Students, Education, Distance, Internet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review