CoChat
Sign Up
Cited by 40Online Learning and AnalyticsOnline and Blended LearningIntelligent Tutoring Systems and Adaptive Learning

Automated Educative Nudges to Reduce Missed Assignments in College

Open access
Motz, Benjamin A., Mallon, Matthew G., Quick, Joshua D.·2021·IEEE Transactions on Learning Technologies
DOI 10.1109/tlt.2021.3064613

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

As institutions of higher education increasingly utilize online learning management systems, college students are asked to submit more assignments online. Under this regime, when most assignments are posted and submitted online, it is possible to know if a student is missing a submission for an imminent deadline, and to intervene proactively to reduce missed assignments and improve student outcomes. Toward this goal, we designed and evaluated a scalable targeted intervention: a mobile app that would deploy push notifications when students were missing submissions for assignments with imminent deadlines. Results from two experimental pilots demonstrate that this intervention system significantly decreased missed assignments compared with control notifications about instructor announcements to the class (in Experiment 1), and improved assignment adherence and course grades compared with courses that were not using the app (in Experiment 2). We discuss the benefits and theoretical implications of this behavioral guide rail, a purely informative proactive intervention to mitigate risk in advance of a negative outcome.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tlt.2021.3064613

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Push notifications supported cognitive presence by reminding students of assignments and directing them to information accessible on their mobile devices.

    “In terms of engagement, Pham et al. (2016) found that engagement increased with the use of push notifications, while Motz et al. (2021) reported that push notifications supported cognitive presence by reminding students of assignments and directing them to information accessible on their mobile devices.”

    From You have a notification: the role of push notifications in shaping students’ engagement, self-regulation and academic procrastination (2025) · cites this paper as Motz et al. (2021)

    In “B. This Study” and “II. EXPERIMENT 1” (page numbers unavailable), the paper describes smartphone push notifications that remind students of approaching assignments and link to assignment details in the app. It does not report that the notifications supported cognitive presence; its discussion instead addresses assignment submissions and prospective memory. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

1 retracted

Flagged first, then by citations

Research works printed but not linked: 48

  1. U. Trautwein, "The homework-achievement relation reconsidered: Dif- ferentiating homework time, homework frequency, and homework effort," Learn. Instruct., vol. 17, no. 3, pp. 372-388, 2007.
  2. J. Gikandi, D. Morrow, and N. Davis, "Online formative assessment in higher education: A review of the literature," Comput. Educ., vol. 57, no. 4, pp. 2333-2351, Dec. 2011.
  3. J. Sweller, J. van Merrienboer, and F. Paas, "Cognitive architecture and instructional design," Educ. Psychol. Rev., vol. 10, no. 3, pp. 251-296, Sep. 1998.
  4. P. Bawa, "Retention in online courses: Exploring issues and solutions- A literature review," Sage Open, vol. 6, no. 1, pp. 1-11, Jan. 2016.
  5. M. Herbert, "Staying the course: A study in online student satisfaction and retention," Online J. Distance Learn. Admin., vol. 9, no. 4, 2006.
  6. R. Barber and M. Sharkey, "Course correction: Using analytics to pre- dict course success," in Proc. 2nd Int. Conf. Learn. Anal. Knowl., Apr. 2012, pp. 259-262.
  7. S. Benartzi et al., "Should governments invest more in nudging?," Psy- chol. Sci., vol. 28, no. 8, pp. 1041-1055, Jun. 2017.
  8. G. Walton, "The new science of wise psychological interventions," Cur- rent Directions Psychol. Sci., vol. 23, no. 1, pp. 73-82, Feb. 2014.
  9. K. A. Bird, B. L. Castleman, J. Goodman, and C. Lamberton, "Nudging at a national scale: Experimental evidence from a FAFSA completion campaign," Nat. Bur. Econ. Res., Cambridge, MA, USA, Working Paper 26158, 2017.
  10. T. Rogers and A. Feller, "Reducing student absences at scale by target- ing parents' misbeliefs," Nat. Human Behav., vol. 2, pp. 335-342, May 2018.
  11. G. D. Saenz, L. Geraci, and R. Tirso, "Improving metacognition: A comparison of interventions," Appl. Cogn. Psychol., vol. 33, no. 5, pp. 918-929, Apr. 2019.
  12. R. A. Lazowski and C. S. Hulleman, "Motivation interventions in educa- tion: A meta-analytic review," Rev. Educ. Res., vol. 86, no. 2, pp. 602- 640, Jun. 2016.
  13. A. S. Bryk, L. M. Gomez, A. Grunow, and P. G. LeMahieu, Learning to Improve: How America's Schools Can Get Better At Getting Better, Cambridge, MA, USA: Harvard Educ. Press, 2015.
  14. D. R. Davis and J. T. Abbitt, "An investigation of the impact of an intervention to reduce academic procrastination using short message service (SMS) technology," J. Interact. Online Learn., vol. 12, no. 3, pp. 78-102, 2013.
  15. W. Humphrey, Jr., D. Laverie, and A. Shields, "Exploring the effects of encouraging student performance with text assignment reminders," J. Marketing Educ., vol. 43, pp. 91-102, Mar. 2019.
  16. H. Kauffman, "A review of predictive factors of student success in and satisfaction with online learning," Res. Learn. Technol., vol. 23, pp. 1-13, Aug. 2015.
  17. C. Herodotou, B. Rienties, M. Hlosta, A. Boroowa, C. Mangafa, and Z. Zdrahal, "The scalable implementation of predictive learning analyt- ics at a distance learning university: Insights from a longitudinal case study," Internet High. Educ., vol. 45, Apr. 2020, Art. no. 100725.
  18. J. Qiu et al., "Modeling and predicting learning behavior in MOOCs," in Proc. 9th ACM Int. Conf. Web Search Data Mining, Feb. 2016, pp. 93-102.
  19. B. A. Motz, J. Quick, N. Schroeder, J. Zook, and M. Gunkel, "The valid- ity and utility of activity logs as a measure of student engagement," in Proc. 9th Int. Conf. Learn. Analytics Knowl., Mar. 2019, pp. 300-309.
  20. T. Gr€ une-Yanoff and R. Hertwig, "Nudge versus boost: How coherent are policy and theory?," Minds Mach., vol. 26, no. 1, pp. 149-183, Mar. 2016.
  21. C. Sunstein, "Why nudge?," in The Politics of Libertarian Paternalism. New Haven, CT, USA: Yale Univ. Press, 2014.
  22. R. Hertwig and T. Gr€ une-Yanoff, "Nudging and boosting: Steering or empowering good decisions," Perspectives Psychol. Sci., vol. 12, no. 6, pp. 973-986, Aug. 2017.
  23. M. McDaniel and G. Einstein, "Strategic and automatic processes in prospective memory retrieval: A multiprocess framework," Appl. Cogn. Psychol., vol. 14, no. 7, pp. S127-S144, 2000.
  24. G. Einstein and M. McDaniel, "Prospective memory: Multiple retrieval processes," Current Directions Psychol. Sci., vol. 14, no. 6, pp. 286- 290, Dec. 2005.
  25. O. Vortac, M. Edwards, and C. Manning, "Functions of external cues in prospective memory," Memory, vol. 3, no. 2, pp. 201-219, Jun. 1995.
  26. J. Cash, "Alert fatigue," Amer. J. Health-Syst. Pharmacy, vol. 66, no. 23, pp. 2098-2101, Dec. 2009.
  27. S. Lonn and S. D. Teasley, "Saving time or innovating practice: Investi- gating perceptions and uses of learning management systems," Comput. Educ., vol. 53, no. 3, pp. 686-694, Nov. 2009.
  28. J. Kruschke and T. Liddell, "Bayesian data analysis for newcomers," Psychonomic Bull. Rev., vol. 25, no. 1, pp. 155-177, Feb. 2018.
  29. M. Plummer, "JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling," in Proc. 3rd Int. Workshop Distrib. Stat- ist. Comput., Mar. 2003, pp. 1-10.
  30. M. Denwood, "runjags: An r package providing interface utilities, model templates, parallel computing methods and additional distributions for MCMC models in JAGS," J. Statist. Softw., vol. 71, pp. 1-25, Jul. 2016.
  31. J. Kruschke, Doing Bayesian Data Analysis: A Tutorial With R, JAGS, and Stan. 2nd ed., Cambridge, MA, USA: Academic, 2014.
  32. N. Patsopoulos, "A pragmatic view on pragmatic trials," Dialogues Clin. Neurosci., vol. 13, no. 2, pp. 217-224, Jun. 2011.
  33. M. Hoffman and A. Gelman, "The no-u-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo," J. Mach. Learn. Res., vol. 15, no. 1, pp. 1593-1623, Jan. 2014.
  34. A. Gelman and D. Rubin, "Inference from iterative simulation using multiple sequences," Statist. Sci., vol. 7, no. 4, pp. 457-472, 1992.
  35. A. Legg and J. Wilson, "E-mail from professor enhances student motiva- tion and attitudes," Teaching Psychol., vol. 36, no. 3, pp. 205-211, Jun. 2009.
  36. J. Eyink, B. Motz, G. Heltzel, and T. Liddell, "Self-regulated studying behavior, and the social norms that influence it," J. Appl. Social Psy- chol., vol. 50, pp. 10-21, Nov. 2019.
  37. A. Pardo et al., "OnTask: Delivering data-informed, personalized learn- ing support actions," J. Learn. Analytics, vol. 5, no. 3, pp. 235-249, Dec. 2018.
  38. F. Muench and A. Baumel, "More than a text message: Dismantling dig- ital triggers to curate behavior change in patient-centered health inter- ventions," J. Med. Internet Res., vol. 19, no. 5, May 2017, Art. no. e147.
  39. J. Harackiewicz and S. Priniski, "Improving student outcomes in higher education: The science of targeted intervention," Annu. Rev. Psychol., vol. 69, no. 1, pp. 409-435, Jan. 2018.
  40. K. Stawarz, A. Cox, and A. Blandford, "Beyond self-tracking and reminders: Designing smartphone apps that support habit formation," in Proc. 33rd Annu. ACM Conf. Human Factors Comput. Syst., Apr. 2015, pp. 2653-2662.
  41. C. Andrews, B. Motz, J. Israel, and H. Leary, "Characteristics of students who opted in to use the boost mobile app as an educational support service," J. Teaching Learn. Technol., vol. 9, no. 1, pp. 158-170, 2020.
  42. D. Hausman, "Efficacious and ethical public paternalism," Rev. Behav. Econ., vol. 5, no. 3/4, pp. 261-280, 2018.
  43. M. Rizzo and D. Whitman, "Little brother is watching you: New pater- nalism on the slippery slopes," Arizona Law Rev., vol. 51, no. 3, pp. 685-739, 2009.
  44. S. Conly, Against Autonomy: Justifying Coercive Paternalism. Cam- bridge, U.K.: Cambridge Univ. Press, 2012.
  45. D. Whitman, Sweating the Small Stuff: Inner-City Schools and the New Paternalism, WA, DC, USA: Thomas B Fordham Inst. Press, 2008.
  46. K. McDermott and K. Nygreen, "Educational new paternalism: Human capital, cultural capital, and the politics of equal opportunity," Peabody J. Educ., vol. 88, no. 1, pp. 84-97, Jan. 2013.
  47. D. Ariely and K. Wertenbroch, "Procrastination, deadlines, and perfor- mance: Self-control by precommitment," Psychol. Sci., vol. 13, no. 3, pp. 219-224, May 2002.
  48. B. Motz, "Principles for the responsible design of automated student support," EDUCAUSE Review, Transforming Higher Ed, Aug. 2019.

Cited by

Sorted by citations

Authors
Motz, B. A., Mallon, M. G., & Quick, J. D.
Affiliations
Indiana University Bloomington (US)
ORCID
Venue
IEEE Transactions on Learning Technologies
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021-04-01
Volume
14
Issue
2
Pages
189-200
ISSN
  • 1939-1382 (ISSN-L)
  • 2372-0050
DOI
10.1109/tlt.2021.3064613
OpenAlex
W3134829240
Cited by
40
References
9
Citations by year
2026: 7 · 2025: 9 · 2024: 8 · 2023: 6 · 2022: 5 · 2021: 3 · 2020: 2
FWCI
4.65
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Learning Technologies (publishedVersion, cc-by)
  • EdArXiv (OSF Preprints) (submittedVersion, cc-by-nc-nd)
  • EdArXiv (OSF Preprints) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › Education › Online and Blended Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
nudging, online learning management systems, push notifications, mobile app intervention, proactive intervention, course grades, student behavior, higher education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review