Examining Predictors of Real-World User Engagement with Self-Guided eHealth Interventions: Analysis of Mobile Apps and Websites Using a Novel Dataset
Open accessPublished version of a preprint
The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The literature suggests that the product design of self-guided electronic health (eHealth) interventions impacts user engagement. Traditional trial settings, however, do not enable the examination of these relationships in real-world use. OBJECTIVE: This study aimed to examine whether the qualities of product design, research evidence, and publicly available data predict real-world user engagement with mobile and Web-based self-guided eHealth interventions. METHODS: This analysis included self-guided mobile and Web-based eHealth interventions available to the public-with their qualities assessed using the Enlight suite of scales. Scales included Usability, Visual Design, User Engagement, Content, Therapeutic Persuasiveness, Therapeutic Alliance, Credibility, and Research Evidence. Behavioral data on real-world usage were obtained from a panel that provides aggregated nonpersonal information on user engagement with websites and mobile apps, based on a time window of 18 months that was set between November 1, 2016 and April 30, 2018. Real-world user engagement variables included average usage time (for both mobile apps and websites) and mobile app user retention 30 days after download. RESULTS: The analysis included 52 mobile apps (downloads median 38,600; interquartile range [IQR] 116,000) and 32 websites (monthly unique visitors median 5689; IQR 30,038). Results point to moderate correlations between Therapeutic Persuasiveness, Therapeutic Alliance, and the 3 user engagement variables (.31≤rs≤.51; Ps≤.03). Visual Design, User Engagement, and Content demonstrated similar degrees of correlation with mobile app engagement variables (.25≤rs≤.49; Ps≤.04) but not with average usage time of Web-based interventions. Positive correlations were also found between the number of reviews on Google Play and average app usage time (r=.58; P<.001) and user retention after 30 days (r=.23; P=.049). Although several product quality ratings were positively correlated with research evidence, the latter was not significantly correlated with real-world user engagement. Hierarchical stepwise regression analysis revealed that either Therapeutic Persuasiveness or Therapeutic Alliance explained 15% to 26% of user engagement variance. Data on Google Play (number of reviews) explained 15% of the variance of mobile app usage time above Enlight ratings; however, publicly available data did not significantly contribute to explaining the variance of the other 2 user-engagement variables. CONCLUSIONS: Results indicate that the qualities of product design predict real-world user engagement with eHealth interventions. The use of real-world behavioral datasets is a novel way to learn about user behaviors, creating new avenues for eHealth intervention research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Self-guided eHealth interventions that create clinician-patient therapeutic alliance and therapeutic persuasiveness are positively correlated with real-world usage of mobile apps and web-based programs [20].”
From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“There is also evidence that applications with better design quality have better research substantiation, but this does not predict real world use and engagement [20].”
From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [20]
In the citing paper
“For example, a real-world analysis of a range of apps and sites available to the public, with varying support, demonstrated that ratings of the therapeutic alliance (measured using the Enlight measure) were associated with increased user engagement [76].”
From The therapeutic alliance in digital mental health interventions for serious mental illnesses: narrative review (2020) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
KAUFMAN, J., BIRMAHER, B., BRENT, D., RAO, U., FLYNN, C., MORECI, P., WILLIAMSON, D., & RYAN, N. (1997) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199707000-00021 · Cited by 10,148
‘Small Changes' to Diet and Physical Activity Behaviors for Weight Management
Hills, A. P., Byrne, N. M., Lindstrom, R., & Hill, J. O. (2013) · Obesity Facts · doi:10.1159/000345030 · Cited by 9,493
Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329
Nielsen, J. (1994) · Conference companion on Human factors in computing systems - CHI '94 · doi:10.1145/259963.260531 · Cited by 2,759
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 1×
“The impact of these interventions, however, is limited by the ability to engage users in therapeutic activities and to support users’ adherence to the therapeutic process [3,4].”
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · cited 2×
“Enlight is the first suite of measures to incorporate behavior change, persuasive design, and therapeutic alliance concepts—concepts that have been found to affect a program’s therapeutic potential [15,16].”
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · cited 2×
“In recent years, several studies have demonstrated that it is feasible to achieve sufficient inter-rater reliability with criteria-based rating scales by using trained raters [35,51].”
Enhancing the explanatory power of usability heuristics
Nielsen, J. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191729 · Cited by 1,439
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397 · cited 1×
“Therefore, Usability should be considered to be a barrier rather than a facilitator of user engagement [48,49].”
Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 2×
“Enlight is the first suite of measures to incorporate behavior change, persuasive design, and therapeutic alliance concepts—concepts that have been found to affect a program’s therapeutic potential [15,16].”
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · cited 1×
“The impact of these interventions, however, is limited by the ability to engage users in therapeutic activities and to support users’ adherence to the therapeutic process [3,4].”
Lobbestael, J., Leurgans, M., & Arntz, A. (2011) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.693 · Cited by 1,124 · cited 1×
“This notion has been acknowledged for decades within the psychological assessment field ([52,53]; George et al, unpublished data, 1985) we hope that as the eHealth evaluation field moves forward, more attention will be paid to the proper use of methods to train and examine evaluators’ work.”
Collins, L. M., Murphy, S. A., & Strecher, V. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.01.022 · Cited by 1,082
Healthcare via Cell Phones: A Systematic Review
Krishna, S., Boren, S. A., & Balas, E. A. (2009) · Telemedicine and e-Health · doi:10.1089/tmj.2008.0099 · Cited by 882
IBM SPSS Statistics 23 Step by Step
George, D., & Mallery, P. (2016) · Routledge eBooks · doi:10.4324/9781315545899 · Cited by 724
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9275 · Cited by 670 · cited 1×
“This finding is congruent with Fleming et al’s study that examined published reports on the real-world uptake of eHealth programs [50].”
Understanding the Impact of Convergent Validity on Research Results
Carlson, K. D., & Herdman, A. O. (2012) · Organizational Research Methods · doi:10.1177/1094428110392383 · Cited by 642 · cited 1×
“These findings suggested that there was sufficient convergent validity, which can be claimed if the correlation coefficient is above .50, although a value above .70 is usually recommended [45].”
A Review of eHealth Interventions for Physical Activity and Dietary Behavior Change
Norman, G. J., Zabinski, M. F., Adams, M. A., Rosenberg, D. E., Yaroch, A. L., & Atienza, A. A. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.05.007 · Cited by 605
A Behavior Change Model for Internet Interventions
Ritterband, L. M., Thorndike, F. P., Cox, D. J., Kovatchev, B. P., & Gonder-Frederick, L. A. (2009) · Annals of Behavioral Medicine · doi:10.1007/s12160-009-9133-4 · Cited by 591 · cited 1×
“There is a body of literature examining how intervention design may facilitate engagement and behavior change [10-14].”
Kazdin, A. E. (2017) · Behaviour Research and Therapy · doi:10.1016/j.brat.2016.06.004 · Cited by 554 · cited 1×
“Self-guided electronic health (eHealth) interventions have the potential to increase access to evidence-based care, while reducing the costs associated with service uptake [1,2].”
Who uses mobile phone health apps and does use matter? A secondary data analytics approach
Carroll, J. K., Moorhead, A., Bond, R., LeBlanc, W. G., Petrella, R. J., & Fiscella, K. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.5604 · Cited by 540
Online Prevention Aimed at Lifestyle Behaviors: A Systematic Review of Reviews
Kohl, L. F., Crutzen, R., & de Vries, N. K. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2665 · Cited by 388
Ly, K. H., Ly, A.-M., & Andersson, G. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.10.002 · Cited by 368
Mohr, D. C., Weingardt, K. R., Reddy, M., & Schueller, S. M. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600541 · Cited by 365 · cited 3×
“…relationship between research evidence and real-world uptake is key in light of the notion that efficacy trials largely emphasize internal validity over real-world issues, such as the technological environment, implementation, and sustainability, and thus may not provide the needed validation [26].”
Intervention Mapping: Protocol for Applying Health Psychology Theory to Prevention Programmes
Kok, G., Schaalma, H., Ruiter, R. A. C., Van Empelen, P., & Brug, J. (2004) · Journal of Health Psychology · doi:10.1177/1359105304038379 · Cited by 362
Atkinson, M., & Kydd, C. (1997) · ACM SIGMIS Database: the DATABASE for Advances in Information Systems · doi:10.1145/264701.264705 · Cited by 356
Emerging mHealth and eHealth interventions for serious mental illness: a review of the literature
Naslund, J. A., Marsch, L. A., McHugo, G. J., & Bartels, S. J. (2015) · Journal of Mental Health · doi:10.3109/09638237.2015.1019054 · Cited by 327 · cited 1×
“As eHealth interventions require individuals to engage with self-care outside of traditional health care settings [5-7], individuals’ engagement must compete with other events in their daily lives and fluctuating motivation to engage in effortful behavior [8].”
Cugelman, B., Thelwall, M., & Dawes, P. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1367 · Cited by 323 · cited 1×
“Data suggest that there is a strong relationship between engagement and efficacy [54-56]; however, more does not always equal better [57].”
Measuring Web application quality with WebQEM
Olsina, L., & Rossi, G. (2002) · IEEE Multimedia · doi:10.1109/mmul.2002.1041945 · Cited by 255
MAINTAINING ENGAGEMENT IN LONG-TERM INTERVENTIONS WITH RELATIONAL AGENTS
Bickmore, T., Schulman, D., & Yin, L. (2010) · Applied Artificial Intelligence · doi:10.1080/08839514.2010.492259 · Cited by 246
Baumel, A., Faber, K., Mathur, N., Kane, J. M., & Muench, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7270 · Cited by 206 · cited 6×
“The quality scores were generated using Enlight [21].”
InfluentialSelf-citationChristensen, H., Griffiths, K., Groves, C., & Korten, A. (2006) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/j.1440-1614.2006.01743.x · Cited by 201 · cited 1×
“Data suggest that there is a strong relationship between engagement and efficacy [54-56]; however, more does not always equal better [57].”
Moller, A. C., Merchant, G., Conroy, D. E., West, R., Hekler, E., Kugler, K. C., & Michie, S. (2017) · Journal of Behavioral Medicine · doi:10.1007/s10865-016-9818-7 · Cited by 151 · cited 1×
“The big data commonly generated and stored by digital platforms can be used to learn about user behaviors in order to refine conceptual models or theories, and to understand processes related to eHealth interventions [24].”
A Systematic Review of Quality Assessment Methods for Smartphone Health Apps
BinDhim, N. F., Hawkey, A., & Trevena, L. (2015) · Telemedicine and e-Health · doi:10.1089/tmj.2014.0088 · Cited by 142
Muench, F., & Baumel, A. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7463 · Cited by 138
Self-citationStatistics for the Behavioral Sciences
Privitera, G. J. (2011) · Cited by 132
Baumel, A., & Yom-Tov, E. (2018) · Translational Behavioral Medicine · doi:10.1093/tbm/ibx037 · Cited by 128 · cited 5×
“Similar to previous findings, Visual Design, User Engagement, and Content were not found to have a significant correlation with usage time of Web-based interventions [37].”
InfluentialSelf-citationPowell, A. C., Torous, J., Chan, S., Raynor, G. S., Shwarts, E., Shanahan, M., & Landman, A. B. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5176 · Cited by 101
(Inter)personal Computing: The Role of the Therapeutic Relationship in E-mental Health
Cavanagh, K., & Millings, A. (2013) · Journal of Contemporary Psychotherapy · doi:10.1007/s10879-013-9242-z · Cited by 88
Provost, M., Koompalum, D., Dong, D., & Martin, B. C. (2005) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2005.07.034 · Cited by 86
Makai, P., Perry, M., Robben, S. H., Schers, H. J., Heinen, M. M., Olde Rikkert, M. G., & Melis, R. F. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3057 · Cited by 59 · cited 1×
“Data suggest that there is a strong relationship between engagement and efficacy [54-56]; however, more does not always equal better [57].”
Heuristic evaluation of persuasive health technologies
Kientz, J. A., Choe, E. K., Birch, B., Maharaj, R., Fonville, A., Glasson, C., & Mundt, J. (2010) · International Health Informatics Symposium · doi:10.1145/1882992.1883084 · Cited by 53
Mediators of a successful web‐based smokeless tobacco cessation program
Danaher, B. G., Smolkowski, K., Seeley, J. R., & Severson, H. H. (2008) · Addiction · doi:10.1111/j.1360-0443.2008.02295.x · Cited by 45
Dobbins, M., Watson, S., Read, K., Graham, K., Nooraie, R. Y., & Levinson, A. J. (2018) · JMIR Aging · doi:10.2196/aging.9216 · Cited by 38
Baumel, A., Birnbaum, M. L., & Sucala, M. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0776-6 · Cited by 36 · cited 1×
“Enlight is a comprehensive suite of criteria-based measurements—completed by trained raters who review the eHealth program—that was developed by a multidisciplinary team through a rigorous process of content analysis, grouping, and classification of 476 identified criteria items [21,40].”
Helen, M., Janessa, G., & Kushniruk Andre, W. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-358 · Cited by 29
Internet-Based Self-help Interventions for Depression in Routine Care
Ebert, D. D., & Baumeister, H. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.1394 · Cited by 29 · cited 2×
“Additionally, using commonly generated data enables us to record real-world user behaviors outside of study settings that involve interactions with research staff—interactions that might impact user engagement [25,26].”
How a Fully Automated eHealth Program Simulates Three Therapeutic Processes: A Case Study
Holter, M. T. S., Johansen, A., & Brendryen, H. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5415 · Cited by 28
Baumel, A., Baker, J. T., Birnbaum, M. L., Christensen, H. M., Choudhury, M. D., Mohr, D. C., Muench, F. J., Schlosser, D. A., Titov, N., & Kane, J. M. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700270 · Cited by 17
Self-citation
Research works printed but not linked: 2
- 10.1111/j.1751-9004.2007.00001 · doi:10.1111/j.1751-9004.2007.00001
- Madan, A, 2012, Int J Eng Sci, 4, 590
Cited by
Objective user engagement with mental health apps: systematic search and panel-based usage analysis
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019) · Journal of Medical Internet Research · doi:10.2196/14567 · Cited by 965
“Relational factors have also been previously acknowledged to nurture a therapeutic alliance with users [45-47], and have demonstrated to be a quality aspect that predicts user engagement with mobile health interventions [28].”
Self-citationBeatty, C., Malik, T., Meheli, S., & Sinha, C. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.847991 · Cited by 224
Tremain, H., McEnery, C., Fletcher, K., & Murray, G. (2020) · JMIR Mental Health · doi:10.2196/17204 · Cited by 193
Baumel, A., Edan, S., & Kane, J. M. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz147 · Cited by 143
Self-citationGamifying CBT to deliver emotional health treatment to young people on smartphones
Christie, G. I., Shepherd, M., Merry, S. N., Hopkins, S., Knightly, S., & Stasiak, K. (2019) · Internet Interventions · doi:10.1016/j.invent.2019.100286 · Cited by 81
Metadata
- Authors
- Baumel, A., & Kane, J. M.
- Affiliations
- University of Haifa (IL)
- Zucker Hillside Hospital (US)
- Corresponding author
- Baumel, Amit
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-12-14
- Volume
- 20
- Issue
- 12
- Pages
- e11491
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6315225
- OpenAlex
- W2904531033
- Cited by
- 131
- References
- 50
- Citations by year
- 2026: 12 · 2025: 17 · 2024: 18 · 2023: 25 · 2022: 14 · 2021: 16 · 2020: 18 · 2019: 11
- FWCI
- 17.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- usage data, mobile application, online intervention, therapeutic relationship, product design, usability, visual design, app retention, Google Play reviews, research evidence, credibility, mental health apps
- MeSH terms
- Humans, Persuasive Communication, Self-Help Groups, Telemedicine, Mobile Applications, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
