To Prompt or Not to Prompt? A Microrandomized Trial of Time-Varying Push Notifications to Increase Proximal Engagement With a Mobile Health App
Open accessPublished version of a preprint
The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Mobile health (mHealth) apps provide an opportunity for easy, just-in-time access to health promotion and self-management support. However, poor user engagement with these apps remains a significant unresolved challenge. OBJECTIVE: This study aimed to assess the effect of sending versus not sending a push notification containing a contextually tailored health message on proximal engagement, measured here as self-monitoring via the app. Secondary aims were to examine whether this effect varies by the number of weeks enrolled in the program or by weekday versus weekend. An exploratory aim was to describe how the effect on proximal engagement differs between weekday versus weekend by the time of day. METHODS: The study analyzes the causal effects of push notifications on proximal engagement in 1255 users of a commercial workplace well-being intervention app over 89 days. The app employs a microrandomized trial (MRT) design to send push notifications. At 1 of 6 times per day (8:30 am, 12:30 pm, 5:30 pm, 6:30 pm, 7:30 pm, and 8:30 pm; selected randomly), available users were randomized with equal probability to be sent or not sent a push notification containing a tailored health message. The primary outcome of interest was whether the user self-monitored behaviors and feelings at some time during the next 24 hours via the app. A generalization of log-linear regression analysis, adapted for use with data arising from an MRT, was used to examine the effect of sending a push notification versus not sending a push notification on the probability of engagement over the next 24 hours. RESULTS: Users were estimated to be 3.9% more likely to engage with the app in the next 24 hours when a tailored health message was sent versus when it was not sent (risk ratio 1.039; 95% CI 1.01 to 1.08; P<.05). The effect of sending the message attenuated over the course of the study, but this effect was not statistically significant (P=.84). The effect of sending the message was greater on weekends than on weekdays, but the difference between these effects was not statistically significant (P=.18). When sent a tailored health message on weekends, the users were 8.7% more likely to engage with the app (95% CI 1.01 to 1.17), whereas on weekdays, the users were 2.5% more likely to engage with the app (95% CI 0.98 to 1.07). The effect of sending a tailored health message was greatest at 12:30 pm on weekends, when the users were 11.8% more likely to engage (90% CI 1.02 to 1.13). CONCLUSIONS: Sending a push notification containing a tailored health message was associated with greater engagement in an mHealth app. Results suggested that users are more likely to engage with the app within 24 hours when push notifications are sent at mid-day on weekends.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is already some promising research being done in this space, with several studies finding that external strategies such as automated reminders (13, 67), therapist-led coaching (68, 69), and moderated peer-support groups (70) can be effective toward promoting engagement with digital interventions for mental health.”
From Effect of Engagement With Digital Interventions on Mental Health Outcomes: A Systematic Review and Meta-Analysis (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Low engagement—defined as suboptimal levels of user access and/or adherence to an intervention (12)—is touted as one of the main reasons why the potential benefits of these interventions remain unrealized in the real world (13, 14).”
From Effect of Engagement With Digital Interventions on Mental Health Outcomes: A Systematic Review and Meta-Analysis (2021) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reminders [37-40,43,46,48,51,52,54,56-58,62,66,67,69-71] to engage with an app were reported to be useful for people with busy schedules and for those who tend to forget engaging with the app and, therefore, with the target behavior [37,39,43,56,67].”
From Influences on the Uptake of and Engagement With Health and Well-Being Smartphone Apps: Systematic Review (2020) · cites this paper as [43]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Under capability, the main factors identified were app literacy skills (B), app awareness (U), available user guidance (B), health information (E), statistical information on progress (E), well-designed reminders (E), features to reduce cognitive load (E), and self-monitoring features (E).”
From Influences on the Uptake of and Engagement With Health and Well-Being Smartphone Apps: Systematic Review (2020) · cites this paper as [43]
The cited source says what the claim attributes to it.
In the citing paper
“A microrandomized trial found that a push notification that contained a tailored health message resulted in a small increase in the engagement with a health app [43].”
From Influences on the Uptake of and Engagement With Health and Well-Being Smartphone Apps: Systematic Review (2020) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 1×
“In fact, traditional Web-based interventions observe decreases in engagement over time, with large proportions of participants dropping out or discontinuing the use of the app completely [13].”
BackgroundWhat is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397
Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 1×
“Engagement could also be the subjective quality of user experiences with the app, which can be influenced by design elements [16].”
BackgroundSelf-monitoring in weight loss: a systematic review of the literature
Burke, L. E., Wang, J., & Sevick, M. A. (2011) · Journal of the American Dietetic Association · doi:10.1016/j.jada.2010.10.008 · Cited by 1,203 · cited 1×
“In published studies of different behavioral interventions, self-monitoring is encouraged at a variety of frequencies (up to multiple times per day) [2,12,47-49].”
BackgroundUnderstanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118
Understanding tailoring in communicating about health
Hawkins, R. P., Kreuter, M., Resnicow, K., Fishbein, M., & Dijkstra, A. (2008) · Health Education Research · doi:10.1093/her/cyn004 · Cited by 934 · cited 1×
“The messages were created via tailoring strategies, are personalized, and the content was organized by types to both enhance message processing and interaction with the app [38] (see Figure 1).”
BackgroundDennison, L., Morrison, L., Conway, G., & Yardley, L. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2583 · Cited by 895
Microrandomized trials: An experimental design for developing just-in-time adaptive interventions.
Klasnja, P., Hekler, E. B., Shiffman, S., Boruvka, A., Almirall, D., Tewari, A., & Murphy, S. A. (2015) · Health Psychology · doi:10.1037/hea0000305 · Cited by 671 · cited 8×
“Third, the use of the MRT design to repeatedly randomize users over the course of the trial allowed us to investigate the causal effects of push notifications on proximal engagement and the real-time, real-world conditions that likely influence these proximal effects.”
MethodInfluentialSelf-citationWho uses mobile phone health apps and does use matter? A secondary data analytics approach
Carroll, J. K., Moorhead, A., Bond, R., LeBlanc, W. G., Petrella, R. J., & Fiscella, K. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.5604 · Cited by 540
Mobile Health Apps to Facilitate Self-Care: A Qualitative Study of User Experiences
Anderson, K., Burford, O., & Emmerton, L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0156164 · Cited by 502
Using smartphones and health apps to change and manage health behaviors: a population-based survey
Ernsting, C., Dombrowski, S. U., Oedekoven, M., O´Sullivan, J. L., Kanzler, M., Kuhlmey, A., & Gellert, P. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6838 · Cited by 400 · cited 1×
“Individual characteristics such as age, education levels, and state of health are known to equally influence the adoption of mHealth apps, in addition to factors such as being female, younger, higher education levels, and lower body mass index increasing the odds [17,18].”
BackgroundFirth, J., Cotter, J., Torous, J., Bucci, S., Firth, J. A., & Yung, A. R. (2016) · Schizophrenia Bulletin · doi:10.1093/schbul/sbv132 · Cited by 351 · cited 1×
“In recent years, owing to increasingly ubiquitous smartphone ownership at a population level and rapidly reducing digital divides [4], smartphone apps have emerged as an important channel for health behavior interventions.”
BackgroundAn in-situ study of mobile phone notifications
Pielot, M., Church, K., & de Oliveira, R. (2014) · Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services · doi:10.1145/2628363.2628364 · Cited by 345
Large-scale assessment of mobile notifications
Sahami Shirazi, A., Henze, N., Dingler, T., Pielot, M., Weber, D., & Schmidt, A. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557189 · Cited by 341
Stephenson, A., McDonough, S. M., Murphy, M. H., Nugent, C. D., & Mair, J. L. (2017) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-017-0561-4 · Cited by 290 · cited 1×
“In published studies of different behavioral interventions, self-monitoring is encouraged at a variety of frequencies (up to multiple times per day) [2,12,47-49].”
BackgroundInterruptMe: designing intelligent prompting mechanisms for pervasive applications
Pejovic, V., & Musolesi, M. (2014) · Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2632048.2632062 · Cited by 282
The Effectiveness of Prompts to Promote Engagement With Digital Interventions: A Systematic Review
Alkhaldi, G., Hamilton, F. L., Lau, R., Webster, R., Michie, S., & Murray, E. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4790 · Cited by 273 · cited 1×
“In contrast, the fields of public health and psychology contain several well-designed randomized controlled trial studies to evaluate text-messaging interventions that are similar to push notification interventions in mHealth apps [33,34].”
BackgroundMy Phone and Me: Understanding People's Receptivity to Mobile Notifications
Mehrotra, A., Pejovic, V., Vermeulen, J., Hendley, R., & Musolesi, M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858566 · Cited by 261
"Silence Your Phones": Smartphone Notifications Increase Inattention and Hyperactivity Symptoms
Kushlev, K., Proulx, J., & Dunn, E. W. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858359 · Cited by 245
Bakker, D., & Rickard, N. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.11.016 · Cited by 231
Preventive Health Behavior Change Text Message Interventions: A Meta-analysis
Armanasco, A. A., Miller, Y. D., Fjeldsoe, B. S., & Marshall, A. L. (2017) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.10.042 · Cited by 210 · cited 1×
“Evidence demonstrates that providing engaged individuals with a digital behavior change intervention improves health outcomes [1-3].”
BackgroundMartinez, O., Wu, E., Shultz, A. Z., Capote, J., López Rios, J., Sandfort, T., Manusov, J., Ovejero, H., Carballo-Dieguez, A., Chavez Baray, S., et al. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3311 · Cited by 196
Morton, K., Dennison, L., May, C., Murray, E., Little, P., McManus, R. J., & Yardley, L. (2017) · Patient Education and Counseling · doi:10.1016/j.pec.2016.10.019 · Cited by 195
Self-affirmation alters the brain’s response to health messages and subsequent behavior change.
Falk, E. B., O’Donnell, M. B., Cascio, C. N., Tinney, F., Kang, Y., Lieberman, M. D., Taylor, S. E., An, L., Resnicow, K., & Strecher, V. J. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1500247112 · Cited by 190 · cited 2×
“For example, engagement might be classified as use of the app to self-monitor health behaviors and feelings; self-monitoring has been demonstrated to have a positive impact on well-being and improve health behaviors [14,15].”
BackgroundFischer, J. E., Greenhalgh, C., & Benford, S. (2011) · Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services · doi:10.1145/2037373.2037402 · Cited by 187
Efficacy of SMS Text Message Interventions for Smoking Cessation: A Meta-Analysis
Spohr, S. A., Nandy, R., Gandhiraj, D., Vemulapalli, A., Anne, S., & Walters, S. T. (2015) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2015.01.011 · Cited by 166
Morrison, L. G., Hargood, C., Pejovic, V., Geraghty, A. W. A., Lloyd, S., Goodman, N., Michaelides, D. T., Weston, A., Musolesi, M., Weal, M. J., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0169162 · Cited by 162 · cited 3×
“This is contrary to the lack of effect of the timing of notifications on engagement observed in prior research [35].”
BackgroundResultAnticipatory Mobile Computing: A Survey of the State of the Art and Research Challenges
Pejovic, V., & Musolesi, M. (2015) · ACM Computing Surveys · doi:10.1145/2693843 · Cited by 154
Assessing time-varying causal effect moderation in mobile health
Boruvka, A., Almirall, D., Witkiewitz, K., & Murphy, S. A. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1305274 · Cited by 152
Self-citationAnticipatory Mobile Computing: A Survey of the State of the Art and Research Challenges
Pejović, V., & Musolesi, M. (2013) · Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) · doi:10.48550/arxiv.1306.2356 · Cited by 150
The power of mobile notifications to increase wellbeing logging behavior
Bentley, F., & Tollmar, K. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466140 · Cited by 99
Van Gemert-Pijnen, J. E., Kelders, S. M., & Bohlmeijer, E. T. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2991 · Cited by 84
Morrison, L. G., Hargood, C., Lin, S. X., Dennison, L., Joseph, J., Hughes, S., Michaelides, D. T., Johnston, D., Johnston, M., Michie, S., et al. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3579 · Cited by 79
Turner, L. D., Allen, S. M., & Whitaker, R. M. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2807514 · Cited by 74
Kanejima, Y., Kitamura, M., & Izawa, K. P. (2018) · Aging Clinical and Experimental Research · doi:10.1007/s40520-018-0960-7 · Cited by 71
Characterizing user engagement with health app data: a data mining approach
Serrano, K. J., Coa, K. I., Yu, M., Wolff-Hughes, D. L., & Atienza, A. A. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-017-0508-y · Cited by 67 · cited 1×
“Even among those who use the apps, the amount of use depends on an individual’s health and behavioral characteristics [10-12].”
BackgroundUnderstanding the Role of Places and Activities on Mobile Phone Interaction and Usage Patterns
Mehrotra, A., Müller, S. R., Harari, G. M., Gosling, S. D., Mascolo, C., Musolesi, M., & Rentfrow, P. J. (2017) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3131901 · Cited by 66
Bidargaddi, N., Musiat, P., Winsall, M., Vogl, G., Blake, V., Quinn, S., Orlowski, S., Antezana, G., & Schrader, G. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6775 · Cited by 61
Self-citationRahman, Q. A., Janmohamed, T., Pirbaglou, M., Ritvo, P., Heffernan, J. M., Clarke, H. A., & Katz, J. D. (2017) · JMIR mhealth and uhealth · doi:10.2196/mhealth.7871 · Cited by 52
Characterizing Smartphone Engagement for Schizophrenia: Results of a Naturalist Mobile Health Study
Torous, J., Staples, P., Slaters, L., Adams, J., Sandoval, L., Onnela, J., & Keshavan, M. (2017) · Clinical Schizophrenia & Related Psychoses · doi:10.3371/csrp.jtps.071317 · Cited by 44 · cited 1×
“Even among those who use the apps, the amount of use depends on an individual’s health and behavioral characteristics [10-12].”
BackgroundInvestigating The Role of Task Engagement in Mobile Interruptibility
Pejović, V., Musolesi, M., & Mehrotra, A. (2015) · International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI Adjunct) · doi:10.1145/2786567.2794336 · Cited by 42
Preliminary investigations about interruptibility of smartphone users at specific place types
Exler, A., Braith, M., Schankin, A., & Beigl, M. (2016) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2968219.2968554 · Cited by 41
Push Notifications in Diet Apps: Influencing Engagement Times and Tasks
Freyne, J., Yin, J., Brindal, E., Hendrie, G. A., Berkovsky, S., & Noakes, M. (2017) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2017.1278896 · Cited by 31
Self-monitoring Lifestyle Behavior in Overweight and Obese Pregnant Women: Qualitative Findings
Shieh, C., & Draucker, C. B. (2018) · Clinical Nurse Specialist · doi:10.1097/nur.0000000000000355 · Cited by 13
Zhang, S., & Elhadad, N. (2016) · PubMed · Cited by 10
Dempsey, W., Liao, P., Kumar, S., & Murphy, S. A. (2017) · arXiv · doi:10.48550/arxiv.1711.03587
Self-citation
Research works printed but not linked: 2
- Strecher, VJ, 2016, Life on Purpose
- Zhang, S, 2016, AMIA Annu Symp Proc, 2016, 2090
Cited by
Challenges in Participant Engagement and Retention Using Mobile Health Apps: Literature Review
Amagai, S., Pila, S., Kaat, A. J., Nowinski, C. J., & Gershon, R. C. (2022) · Journal of Medical Internet Research · doi:10.2196/35120 · Cited by 343
“Refer to Figure 1 for the study selection flowchart and Multimedia Appendix 1 [4-8,17-73] for characteristics of the studies.”
BackgroundSzinay, D., Jones, A., Chadborn, T., Brown, J., & Naughton, F. (2020) · Journal of Medical Internet Research · doi:10.2196/17572 · Cited by 289
“The ability to customize reminders Engagement Well-designed reminders [37-40,43,46,48,51,52,54,56-58,62,66-69,71,78]”
BackgroundInfluentialGan, D. Z. Q., McGillivray, L., Han, J., Christensen, H., & Torok, M. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.764079 · Cited by 305
“Low engagement— defined as suboptimal levels of user access and/or adherence to an intervention (12)—is touted as one of the main reasons why the potential benefits of these interventions remain unrealized in the real world (13, 14).”
BackgroundWei, Y., Zheng, P., Deng, H., Wang, X., Li, X., & Fu, H. (2020) · Journal of Medical Internet Research · doi:10.2196/21687 · Cited by 234
Lipschitz, J. M., Pike, C. K., Hogan, T. P., Murphy, S. A., & Burdick, K. E. (2023) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-023-00297-3 · Cited by 225
Self-citation
Metadata
- Authors
- Bidargaddi, N., Almirall, D., Murphy, S., Nahum-Shani, I., Kovalcik, M., Pituch, T., Maaieh, H., & Strecher, V.
- Affiliations
- University of Michigan (US)
- Harvard University (US)
- Corresponding author
- Bidargaddi, Niranjan
- ORCID
- Venue
- JMIR mHealth and uHealth
- Publisher
- JMIR Publications Inc.
- Published
- 2018-11-29
- Volume
- 6
- Issue
- 11
- Pages
- e10123
- ISSN
- 2291-5222 (ISSN-L)
- PMC
- PMC6293241
- OpenAlex
- W2836815702
- Cited by
- 189
- References
- 45
- Citations by year
- 2026: 30 · 2025: 22 · 2024: 24 · 2023: 31 · 2022: 33 · 2021: 22 · 2020: 21 · 2019: 6
- FWCI
- 18.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR mhealth and uhealth (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Flinders Academic Commons (Flinders University) (submittedVersion)
- Flinders Academic Commons (Flinders University) (submittedVersion, cc-by)
- Funders
- National Institute on Alcohol Abuse and Alcoholism (R01AA023187)
- National Institute on Drug Abuse (P50DA039838, R01DA039901)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- push notifications, self-monitoring, tailored health messages, workplace well-being interventions, time of day effects, user retention, digital health interventions, health behavior change, user-generated images, user-generated video
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.