The DOI matches its Crossref record; 35 of the 36 references it cites were found in a registry or an index, and 1 are not checked yet.
Scaling up behavioral science interventions in online education
Open accessAbstract
Significance: Low persistence in educational programs is a major obstacle to social mobility. Scientists have proposed many scalable interventions to support students learning online. In one of the largest international field experiments in education, we iteratively tested established behavioral science interventions and found small benefits depending on individual and contextual characteristics. Forecasting intervention efficacy using state-of-the-art methods yields limited improvements. Online education provides unprecedented access to learning opportunities, as evidenced by its role during the 2020 coronavirus pandemic, but adequately supporting diverse students will require more than a light-touch intervention. Our findings encourage funding agencies and researchers conducting large-scale field trials to consider dynamic investigations to uncover and design for contextual heterogeneity to complement static investigations of overall effects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, asynchronous courses often rely heavily on self- regulation and intrinsic motivation, which may raise challenges for learners, such as low engagement, high dropout rates, and a lack of immediate feedback (Kizilcec etal.2020).”
From Game‐Based Learning for Asynchronous AI Literacy Course: Approach to Improve Students' Cognitive, Behavioural, Affective, and Ethical Learning of AI (2026) · cites this paper as Kizilcec et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This intervention increased completion rates only for students in individual countries.”
From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This second intervention was found to increase completion rates only in a subset of courses carried out in developing countries.”
From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Overall, the authors conclude that the impacts of such interventions could be highly heterogeneous across contexts, type of participants and that support interventions should be targeted and take those factors in due account.”
From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“While this intervention increased shortterm persistence, it did not increase course completion, which was previously found (Yeomans and Reich, 2017).”
From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“However, robust evidence on what works is limited and is not specifically focused on teachers (Kizilcec et al., 2020).”
From Using personalized support & school-based study groups to increase teachers’ participation in MOOCs: Findings from two European pilot projects (2021) · cites this paper as Kizilcec et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Implementation intentions and goal achievement: a meta-analysis of effects and processes
Gollwitzer, P. M., & Sheeran, P. (2006) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(06)38002-1 · Cited by 3,236 · cited 2×
“The “mental contrasting with implementation intentions” (MCII) intervention prompts students to reflect on the benefits and barriers to achieving their goal (20) and then plan ahead for how to overcome obstacles (15).”
BackgroundSurrogate endpoints in clinical trials: Definition and operational criteria
Prentice, R. L. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080407 · Cited by 2,057
A national experiment reveals where a growth mindset improves achievement
Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., et al. (2019) · Nature · doi:10.1038/s41586-019-1466-y · Cited by 1,513
Promoting Interest and Performance in High School Science Classes
Hulleman, C. S., & Harackiewicz, J. M. (2009) · Science · doi:10.1126/science.1177067 · Cited by 1,220
Reducing the racial achievement gap: A social-psychological intervention
Cohen, G. L., Garcia, J., Apfel, N., & Master, A. (2006) · Science · doi:10.1126/science.1128317 · Cited by 1,073
Wise interventions: Psychological remedies for social and personal problems.
Walton, G. M., & Wilson, T. D. (2018) · Psychological Review · doi:10.1037/rev0000115 · Cited by 617
Future thought and behaviour change
Oettingen, G. (2012) · European Review of Social Psychology · doi:10.1080/10463283.2011.643698 · Cited by 504
Site Selection Bias in Program Evaluation *
Allcott, H. (2015) · The Quarterly Journal of Economics · doi:10.1093/qje/qjv015 · Cited by 426
Hall, A. T., Frink, D. D., & Buckley, M. R. (2017) · Journal of Organizational Behavior · doi:10.1002/job.2052 · Cited by 335
Attrition and Achievement Gaps in Online Learning
Kizilcec, R. F., & Halawa, S. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724680 · Cited by 307
Self-citation
Research works printed but not linked: 3
- C. R. Sunstein, R. H. Thaler, Nudge: Improving Deisions About Health, Wealth, and Happiness, (Yale Univeresity Press, New Haven, 2008).
- C. Zhenghao ., Who’s benefiting from MOOCs, and why. Harv. Bus. Rev. 25, 2–8 (2015).
- 10.1353/bsp.2015.0011 · DOI
Cited by
Sorted by citations
Algorithmic fairness in education
Kizilcec, R. F., & Lee, H. (2022) · Routledge eBooks · doi:10.4324/9780429329067-10 · Cited by 339
Self-citationLiu, S., Liu, S., Liu, Z., Peng, X., & Yang, Z. (2022) · Computers & Education · doi:10.1016/j.compedu.2022.104461 · Cited by 269
Kuo, T. M., Tsai, C., & Wang, J. (2021) · The Internet and Higher Education · doi:10.1016/j.iheduc.2021.100819 · Cited by 210
(2021) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 163
Ionescu, C. A., Paschia, L., Nicolau, N. L. G., Stănescu, S. G., Stancescu, V. M. N., Coman, M. D., & Uzlău, M. C. (2020) · Sustainability · doi:10.3390/su12219030 · Cited by 152
Metadata
- Authors
- Kizilcec, R. F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., Turkay, S., Williams, J. J., & Tingley, D.
- Affiliations
- Cornell University (US)
- Massachusetts Institute of Technology (US)
- Harvard University (US)
- Carnegie Mellon University (US)
- Stanford University (US)
- School of Computer Science, Queensland University of Technology, Brisbane City, QLD 4000, Australia;
- University of Toronto (CA)
- Corresponding author
- Kizilcec, René F.
- Reich, Justin
- Yeomans, Michael
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2020-06-15
- Volume
- 117
- Issue
- 26
- Pages
- 14900-14905
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC7334459
- OpenAlex
- W3034223452
- Cited by
- 208
- References
- 35
- Citations by year
- 2026: 23 · 2025: 30 · 2024: 28 · 2023: 30 · 2022: 39 · 2021: 49 · 2020: 8
- FWCI
- 28.94
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- QUT ePrints (Queensland University of Technology) (submittedVersion, cc-by-nc-nd)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion, cc-by-nc-nd)
- Funders
- National Science Foundation (1646978)
- Stanford University (n/a)
- Microsoft (n/a)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Education › Online and Blended Learning
- Social Sciences › Education › Higher Education Research Studies
- Keywords
- online learning, student persistence, course completion, self-regulation interventions, scaling up interventions, preregistration, MOOCs
- MeSH terms
- Behavior, Behavioral Sciences, Goals, Humans, Research, Students, Education, Distance, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
