The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
The Effect of Timing and Frequency of Push Notifications on Usage of a Smartphone-Based Stress Management Intervention: An Exploratory Trial
Open accessAbstract
Push notifications offer a promising strategy for enhancing engagement with smartphone-based health interventions. Intelligent sensor-driven machine learning models may improve the timeliness of notifications by adapting delivery to a user's current context (e.g. location). This exploratory mixed-methods study examined the potential impact of timing and frequency on notification response and usage of Healthy Mind, a smartphone-based stress management intervention. 77 participants were randomised to use one of three versions of Healthy Mind that provided: intelligent notifications; daily notifications within pre-defined time frames; or occasional notifications within pre-defined time frames. Notification response and Healthy Mind usage were automatically recorded. Telephone interviews explored participants' experiences of using Healthy Mind. Participants in the intelligent and daily conditions viewed (d = .47, .44 respectively) and actioned (d = .50, .43 respectively) more notifications compared to the occasional group. Notification group had no meaningful effects on percentage of notifications viewed or usage of Healthy Mind. No meaningful differences were indicated between the intelligent and non-intelligent groups. Our findings suggest that frequent notifications may encourage greater exposure to intervention content without deterring engagement, but adaptive tailoring of notification timing does not always enhance their use. Hypotheses generated from this study require testing in future work. TRIAL REGISTRATION NUMBER: ISRCTN67177737.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The studies that ranked high for selective reporting bias tended to report only user satisfaction data and treatment adherence, despite their methods considering the collection and assessment of psychiatric scale data and other effectiveness outcomes (29, 76, 79, 85–88, 92).”
From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [79]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In terms of allegiance bias, because of the scarcity and novelty of AI-augmented interventions and other AI algorithms, most of the included articles developed and studied their own AI tools, instead of AI tools developed by other researchers (29, 33, 73, 74, 76, 79, 80, 82–87, 90).”
From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [79]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They found that compared to occasional notifications, intelligent notifications and daily notifications encouraged a higher exposure to the therapeutic content while maintaining engagement levels. However, they noted that the timing of the notification did not seem to support treatment engagement, when comparing intelligent notifications against daily notifications (79).”
From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [79]
The cited source says what the claim attributes to it.
In the citing paper
“In terms of improvement to the delivery of online therapy, the included studies [n=4 (30, 75, 79, 80)] have reported on the implementation of intelligent notifications, momentary sensing, monitoring of treatment outcomes, and triage decision support.”
From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [79]
The cited source says what the claim attributes to it.
In the citing paper
“Though, in the case of Morrison et al. (2017) they found that there was no significant improvement comparing daily to intelligent notifications (79, 80).”
From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [79]
The cited source says what the claim attributes to it.
In the citing paper
“In terms of treatment engagement support, Burns et al. (2011) and Morrison et al. (2017) implemented an AI algorithm that could learn from the user and their environment, to deliver intelligent treatment reminders (79, 80).”
From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [79]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“12 of the included studies (41, 73–83) presented data on treatment dropout, online or remote psychotherapy interventions reported a dropout rate between 0% and 61%, psychoeducation a dropout rate between 4% and 60% and no intervention a dropout rate between 5% and 34%.”
From Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review (2024) · cites this paper as [79]
Experimental — accuracy not yet measuredIn the citing paper
“There were a large variety of study types among the 17 studies, including pilot randomised controlled trials (RCTs) [28–30], observational studies [31–35], model evaluations [36,37], and a qualitative study [38].”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [28]
In the citing paper
“or to personalise aspects of the app such as notification delivery [28]”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [28]
In the citing paper
“12 of the 17 included studies had sample sizes smaller than 100 [28–31,33–35,37–39,41,42].”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [28]
In the citing paper
“predicting or helping manage stress [28,32,33]”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [28]
In the citing paper
“One study using AI/ML to send context-specific notifications found no difference between the ‘intelligent’ notifications and non-intelligent notifications [28].”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [28]
In the citing paper
“Only three randomised controlled trials of an AI-enabled mental health app were identified in this review [28–30]”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [28]
In the citing paper
“All of the studies reviewed used data collected via mobile devices to inform AI models—for natural language analysis and output [29,30,38,40,42,43], risk stratification and prediction [31,41] or for personalising the user’s app experience [28,44].”
From Artificial intelligence and machine learning in mobile apps for mental health: A scoping review (2022) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Tong, A., Sainsbury, P., & Craig, J. (2007) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzm042 · Cited by 42,509
Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548
Sontrop, J. M., Dixon, S. N., Garg, A. X., Jiménez, I. B., Dohein, O., Huang, S. S., & Clark, W. F. (2013) · American Journal of Nephrology · doi:10.1159/000350377 · Cited by 5,903
A behavior model for persuasive design
Fogg, B. (2009) · Proceedings of the 4th International Conference on Persuasive Technology · doi:10.1145/1541948.1541999 · Cited by 2,328
Confidence intervals rather than P values: estimation rather than hypothesis testing.
Gardner, M. J., & Altman, D. G. (1986) · BMJ · doi:10.1136/bmj.292.6522.746 · Cited by 2,014
Yardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442
Self-citationDennison, L., Morrison, L., Conway, G., & Yardley, L. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2583 · Cited by 895
Self-citationMohr, D. C., Burns, M. N., Schueller, S. M., Clarke, G., & Klinkman, M. (2013) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2013.03.008 · Cited by 693 · cited 1×
“Notifications can also prompt access to more intensive support provided by other platforms [12,13].”
BackgroundA meta-analysis of web-delivered tailored health behavior change interventions
Lustria, M. L. A., Noar, S. M., Cortese, J., Van Stee, S. K., Glueckauf, R. L., & Lee, J. (2013) · Journal of Health Communication · doi:10.1080/10810730.2013.768727 · Cited by 689
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655
Nahum-Shani, I., Hekler, E. B., & Spruijt-Metz, D. (2015) · Health Psychology · doi:10.1037/hea0000306 · Cited by 568
The Jigsaw continuous sensing engine for mobile phone applications
Lu, H., Yang, J., Liu, Z., Lane, N. D., Choudhury, T., & Campbell, A. T. (2010) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/1869983.1869992 · Cited by 561
Mobile Health Apps to Facilitate Self-Care: A Qualitative Study of User Experiences
Anderson, K., Burford, O., & Emmerton, L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0156164 · Cited by 502
Brouwer, W., Kroeze, W., Crutzen, R., de Nooijer, J., de Vries, N. K., Brug, J., & Oenema, A. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1639 · Cited by 455
Fry, J. P., & Neff, R. A. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1138 · Cited by 447
Online Prevention Aimed at Lifestyle Behaviors: A Systematic Review of Reviews
Kohl, L. F., Crutzen, R., & de Vries, N. K. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2665 · Cited by 388
An in-situ study of mobile phone notifications
Pielot, M., Church, K., & de Oliveira, R. C. (2014) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/2628363.2628364 · Cited by 383
Predicting human interruptibility with sensors
Fogarty, J. A., Hudson, S. E., Atkeson, C. G., Avrahami, D., Forlizzi, J., Kiesler, S. B., Lee, J. C., & Yang, J. (2005) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1057237.1057243 · Cited by 346
Morrison, L. G., Yardley, L., Powell, J., & Michie, S. (2012) · Telemedicine and e-Health · doi:10.1089/tmj.2011.0062 · Cited by 290
Self-citationPejović, V., & Musolesi, M. (2014) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2632048.2632062 · Cited by 283
Fischer, J. E., Greenhalgh, C., & Benford, S. (2011) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/2037373.2037402 · Cited by 189
Pielot, M., de Oliveira, R., Kwak, H., & Oliver, N. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2556973 · Cited by 187
Designing content-driven intelligent notification mechanisms for mobile applications
Mehrotra, A., Musolesi, M., Hendley, R., & Pejović, V. (2015) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2750858.2807544 · Cited by 186
Strategies for mHealth Research: Lessons from 3 Mobile Intervention Studies
Ben‐Zeev, D., Schueller, S. M., Begale, M. J., Duffecy, J. L., Kane, J. M., & Mohr, D. C. (2014) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0556-2 · Cited by 181
Challenges and opportunities for promoting physical activity in the workplace
Marshall, A. L. (2004) · Journal of science and medicine in sport · doi:10.1016/s1440-2440(04)80279-2 · Cited by 164
How behavioral science can advance digital health
Pagoto, S., & Bennett, G. G. (2013) · Translational Behavioral Medicine · doi:10.1007/s13142-013-0234-z · Cited by 159
Effects of content and time of delivery on receptivity to mobile interruptions
Fischer, J. E., Yee, N., Bellotti, V. M. E., Good, N., Benford, S., & Greenhalgh, C. (2010) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/1851600.1851620 · Cited by 155
Sarker, H., Sharmin, M., Ali, A. A., Rahman, M. M., Bari, R., Hossain, S. M., & Kumar, S. (2014) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2632048.2636082 · Cited by 129
Danaher, B. G., Brendryen, H., Seeley, J. R., Tyler, M. S., & Woolley, T. (2015) · Internet Interventions · doi:10.1016/j.invent.2015.01.002 · Cited by 102
The power of mobile notifications to increase wellbeing logging behavior
Bentley, F. R., & Tollmar, K. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466140 · Cited by 100
Characterizing Periodic Messaging Interventions Across Health Behaviors and Media: Systematic Review
León, E. D., Fuentes, L. W., & Cohen, J. E. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2837 · Cited by 87
Morrison, L. G., Hargood, C., Lin, S. X., Dennison, L., Joseph, J., Hughes, S., Michaelides, D. T., Johnston, D., Johnston, M., Michie, S., et al. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3579 · Cited by 79
Self-citationRiley, W. T., Serrano, K. J., Nilsen, W., & Atienza, A. A. (2015) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2015.03.024 · Cited by 76
Bardus, M., Blake, H., Lloyd, S., & Suzanne Suggs, L. (2014) · International Journal of Workplace Health Management · doi:10.1108/ijwhm-11-2013-0040 · Cited by 54
Self-citationFukuoka, Y., Lindgren, T., & Jong, S. (2012) · Public Health Nursing · doi:10.1111/j.1525-1446.2011.00997.x · Cited by 49
Braun, T., Bambra, C. L., Booth, M., Adetayo, K., & Milne, E. M. G. (2014) · Journal of Public Health · doi:10.1093/pubmed/fdu043 · Cited by 45
Open source smartphone libraries for computational social science
Lathia, N., Rachuri, K. K., Cecilia Mascolo, C., & Roussos, G. (2013) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2494091.2497345 · Cited by 40
Willoughby, J. F., & Furberg, R. (2015) · Journal of Health Communication · doi:10.1080/10810730.2014.977468 · Cited by 37
Suggs, L. S., Blake, H., Bardus, M., & Lloyd, S. (2013) · Journal of Health Services Research & Policy · doi:10.1177/1355819613478001 · Cited by 20
Interest in behaviour change interventions: A conceptual model
Crutzen, R., & Ruiter, R. A. C. (2015) · Research Publications (Maastricht University) · Cited by 13
Flagged first, then by citations
Research works printed but not linked: 1
- Hargood C, Michaelides D, Weal M, Pejovic V, Musolesi M, Morrison L, et al. The UBhave framework: Developing dynamic mobile applications for digital behavioural interventions. Poster presented at: 11th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services; 2014 Dec 02–05; London, UK.
Cited by
Sorted by citations
Applications of Machine Learning in Real-Life Digital Health Interventions: Review of the Literature
Triantafyllidis, A. K., & Tsanas, A. (2019) · Journal of Medical Internet Research · doi:10.2196/12286 · Cited by 243
Fineberg, N. A., Menchón, J. M., Hall, N., Dell’Osso, B. M., Brand, M., Potenza, M. N., Chamberlain, S. R., Cirnigliaro, G., Löchner, C., Billieux, J., et al. (2022) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2022.152346 · Cited by 276
khan, Z. F., & Alotaibi, S. R. (2020) · Journal of Healthcare Engineering · doi:10.1155/2020/8894694 · Cited by 235
Bidargaddi, N., Almirall, D., Murphy, S., Nahum-Shani, I., Kovalcik, M., Pituch, T., Maaieh, H., & Strecher, V. (2018) · JMIR mHealth and uHealth · doi:10.2196/10123 · Cited by 189
Engagement in digital interventions.
Nahum-Shani, I., Shaw, S. D., Carpenter, S. M., Murphy, S. A., & Yoon, C. (2022) · American Psychologist · doi:10.1037/amp0000983 · Cited by 177
Metadata
- Authors
- Morrison, L. G., Hargood, C., Pejovic, V., Geraghty, A. W. A., Lloyd, S., Goodman, N., Michaelides, D. T., Weston, A., Musolesi, M., Weal, M. J., & Yardley, L.
- Affiliations
- University of Southampton (GB)
- Bournemouth University (GB)
- Durham University (GB)
- Gateshead Council (GB)
- University College London (GB)
- Corresponding author
- Morrison, Leanne G.
- ORCID
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-01-03
- Volume
- 12
- Issue
- 1
- Pages
- e0169162
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5207732
- OpenAlex
- W1771486365
- Cited by
- 162
- References
- 41
- Citations by year
- 2026: 13 · 2025: 19 · 2024: 18 · 2023: 15 · 2022: 20 · 2021: 28 · 2020: 19 · 2019: 16 · 2018: 9 · 2017: 3
- FWCI
- 14.68
- Type
- journal article
- Language
- en
- Update notices
- correction (2018-05-22)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Teesside University Research Portal (Teesside University) (publishedVersion, cc-by)
- Explore Bristol Research (publishedVersion, cc-by)
- Bournemouth University Research Online (Bournemouth University) (acceptedVersion)
- ePrints Soton (University of Southampton) (acceptedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by)
- TeesRep (Teesside University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/I032673/1)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Clinical Psychology › Mindfulness and Compassion Interventions
- Keywords
- push notifications, notification timing, smartphone-based intervention, stress management, healthy mind, user engagement, context-aware notifications, mobile health
- MeSH terms
- Smartphone, Machine Learning, Adolescent, Adult, Algorithms, Automation, Female, United Kingdom, Health Promotion, Humans, Male, Middle Aged, Public Health, Quality of Life, Stress, Psychological, Health Behavior, Telemedicine, Geographic Information Systems, Young Adult, Text Messaging, Accelerometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
