CoChat
Sign Up
Cited by 208Online Learning and AnalyticsOnline and Blended LearningHigher Education Research Studies

Scaling up behavioral science interventions in online education

Open access
Kizilcec, René F., Reich, Justin, Yeomans, Michael, Dann, Christoph, Brunskill, Emma, Lopez, Glenn, Turkay, Selen, Williams, Joseph Jay +1 more·2020·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1921417117PMC PMC7334459
Open this paper in CoChat — free accountOpen PDF

Abstract

Significance: Low persistence in educational programs is a major obstacle to social mobility. Scientists have proposed many scalable interventions to support students learning online. In one of the largest international field experiments in education, we iteratively tested established behavioral science interventions and found small benefits depending on individual and contextual characteristics. Forecasting intervention efficacy using state-of-the-art methods yields limited improvements. Online education provides unprecedented access to learning opportunities, as evidenced by its role during the 2020 coronavirus pandemic, but adequately supporting diverse students will require more than a light-touch intervention. Our findings encourage funding agencies and researchers conducting large-scale field trials to consider dynamic investigations to uncover and design for contextual heterogeneity to complement static investigations of overall effects.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1921417117

Cites

35 of 36 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Asynchronous courses often rely heavily on self-regulation and intrinsic motivation, which may raise challenges for learners, such as low engagement, high dropout rates, and a lack of immediate feedback.

    “However, asynchronous courses often rely heavily on self- regulation and intrinsic motivation, which may raise challenges for learners, such as low engagement, high dropout rates, and a lack of immediate feedback (Kizilcec etal.2020).”

    From Game‐Based Learning for Asynchronous AI Literacy Course: Approach to Improve Students' Cognitive, Behavioural, Affective, and Ethical Learning of AI (2026) · cites this paper as Kizilcec et al. 2020

    The Results section (‘Planning Supports Short-Term Persistence’; page unavailable) says online education requires sustained effort and examines engagement and course completion. The Discussion (page unavailable) describes the studied MOOC students as motivated and self-directed. But the paper does not establish that asynchronous courses generally rely on intrinsic motivation or lack immediate feedback, nor does it link those features to dropout. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The mental contrasting with implementation intentions intervention increased completion rates only for students in individual countries.

    “This intervention increased completion rates only for students in individual countries.”

    From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020

    The Interventions and Prior Results section reports that earlier MCII trials increased completion for students in individualistic countries. But the Results section says this effect did not replicate in the present study, and an exploratory analysis found an increase for a different subgroup. The claim omits the prior-study qualification and overstates the effect as exclusive to that group. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The value-relevance intervention increased completion rates only in a subset of courses carried out in developing countries.

    “This second intervention was found to increase completion rates only in a subset of courses carried out in developing countries.”

    From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020

    The Results section reports increased completion for students in less-developed countries, but only in courses with a global achievement gap. The claim incorrectly describes the courses, rather than the students, as being in developing countries; our copy provides no page numbers. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Impacts of such interventions could be highly heterogeneous across contexts and participant types, and support interventions should be targeted.

    “Overall, the authors conclude that the impacts of such interventions could be highly heterogeneous across contexts, type of participants and that support interventions should be targeted and take those factors in due account.”

    From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020

  • The plan making intervention increased short-term persistence but did not increase course completion.

    “While this intervention increased shortterm persistence, it did not increase course completion, which was previously found (Yeomans and Reich, 2017).”

    From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. [1]Sunstein C. R., Thaler R. H., Nudge: Improving Deisions About Health, Wealth, and Happiness, (Yale Univeresity Press, New Haven, 2008).
  2. [10]Napier A., Huttner-Loan E., Reich J., Evaluating transfer of learning from MOOCs to workplaces: A case study from teacher education and Launching Innovation in Schools. Revista Iberoamericana de Educación a Distancia 23 (2), 45– 64 (2020).
  3. [12]Yeomans M., Reich J., “ Planning to learn: Planning prompts encourage and forecast goal pursuit in online education” in Proceedings of the Seventh International Conference on Learning Analytics & Knowledge, (Association for Computing Machinery, 2017), pp. 464– 473.

Cited by

Sorted by citations

Related

Authors
Kizilcec, R. F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., Turkay, S., Williams, J. J., & Tingley, D.
Affiliations
  • Cornell University (US)
  • Massachusetts Institute of Technology (US)
  • Harvard University (US)
  • Carnegie Mellon University (US)
  • Stanford University (US)
  • School of Computer Science, Queensland University of Technology, Brisbane City, QLD 4000, Australia;
  • University of Toronto (CA)
Corresponding author
  • Kizilcec, René F.
  • Reich, Justin
  • Yeomans, Michael
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2020-06-15
Volume
117
Issue
26
Pages
14900-14905
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1921417117
PMC
PMC7334459
OpenAlex
W3034223452
Cited by
208
References
35
Citations by year
2026: 23 · 2025: 30 · 2024: 28 · 2023: 30 · 2022: 39 · 2021: 49 · 2020: 8 · 2019: 1
FWCI
28.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • QUT ePrints (Queensland University of Technology) (submittedVersion, cc-by-nc-nd)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion, cc-by-nc-nd)
Funders
  • National Science Foundation (1646978)
  • Stanford University (n/a)
  • Microsoft (n/a)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › Education › Online and Blended Learning
  • Social Sciences › Education › Higher Education Research Studies
Keywords
online learning, student persistence, course completion, self-regulation interventions, scaling up interventions, preregistration, MOOCs
MeSH terms
Behavior, Behavioral Sciences, Goals, Humans, Research, Students, Education, Distance, Internet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review