Examining Predictors of Real-World User Engagement with Self-Guided eHealth Interventions: Analysis of Mobile Apps and Websites Using a Novel Dataset
Open accessPublished version of a preprint
The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The literature suggests that the product design of self-guided electronic health (eHealth) interventions impacts user engagement. Traditional trial settings, however, do not enable the examination of these relationships in real-world use. OBJECTIVE: This study aimed to examine whether the qualities of product design, research evidence, and publicly available data predict real-world user engagement with mobile and Web-based self-guided eHealth interventions. METHODS: This analysis included self-guided mobile and Web-based eHealth interventions available to the public-with their qualities assessed using the Enlight suite of scales. Scales included Usability, Visual Design, User Engagement, Content, Therapeutic Persuasiveness, Therapeutic Alliance, Credibility, and Research Evidence. Behavioral data on real-world usage were obtained from a panel that provides aggregated nonpersonal information on user engagement with websites and mobile apps, based on a time window of 18 months that was set between November 1, 2016 and April 30, 2018. Real-world user engagement variables included average usage time (for both mobile apps and websites) and mobile app user retention 30 days after download. RESULTS: The analysis included 52 mobile apps (downloads median 38,600; interquartile range [IQR] 116,000) and 32 websites (monthly unique visitors median 5689; IQR 30,038). Results point to moderate correlations between Therapeutic Persuasiveness, Therapeutic Alliance, and the 3 user engagement variables (.31≤rs≤.51; Ps≤.03). Visual Design, User Engagement, and Content demonstrated similar degrees of correlation with mobile app engagement variables (.25≤rs≤.49; Ps≤.04) but not with average usage time of Web-based interventions. Positive correlations were also found between the number of reviews on Google Play and average app usage time (r=.58; P<.001) and user retention after 30 days (r=.23; P=.049). Although several product quality ratings were positively correlated with research evidence, the latter was not significantly correlated with real-world user engagement. Hierarchical stepwise regression analysis revealed that either Therapeutic Persuasiveness or Therapeutic Alliance explained 15% to 26% of user engagement variance. Data on Google Play (number of reviews) explained 15% of the variance of mobile app usage time above Enlight ratings; however, publicly available data did not significantly contribute to explaining the variance of the other 2 user-engagement variables. CONCLUSIONS: Results indicate that the qualities of product design predict real-world user engagement with eHealth interventions. The use of real-world behavioral datasets is a novel way to learn about user behaviors, creating new avenues for eHealth intervention research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Self-guided eHealth interventions that create clinician-patient therapeutic alliance and therapeutic persuasiveness are positively correlated with real-world usage of mobile apps and web-based programs [20].”
From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“There is also evidence that applications with better design quality have better research substantiation, but this does not predict real world use and engagement [20].”
From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [20]
In the citing paper
“For example, a real-world analysis of a range of apps and sites available to the public, with varying support, demonstrated that ratings of the therapeutic alliance (measured using the Enlight measure) were associated with increased user engagement [76].”
From The therapeutic alliance in digital mental health interventions for serious mental illnesses: narrative review (2020) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
KAUFMAN, J., BIRMAHER, B., BRENT, D., RAO, U., FLYNN, C., MORECI, P., WILLIAMSON, D., & RYAN, N. (1997) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199707000-00021 · Cited by 10,148
‘Small Changes' to Diet and Physical Activity Behaviors for Weight Management
Hills, A. P., Byrne, N. M., Lindstrom, R., & Hill, J. O. (2013) · Obesity Facts · doi:10.1159/000345030 · Cited by 9,493
Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999) · Journal of Epidemiology and Community Health · doi:10.1136/jech.53.2.105 · Cited by 3,329
Nielsen, J. (1994) · Conference companion on Human factors in computing systems - CHI '94 · doi:10.1145/259963.260531 · Cited by 2,759
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 1×
“The impact of these interventions, however, is limited by the ability to engage users in therapeutic activities and to support users’ adherence to the therapeutic process [3,4].”
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · cited 2×
“Enlight is the first suite of measures to incorporate behavior change, persuasive design, and therapeutic alliance concepts—concepts that have been found to affect a program’s therapeutic potential [15,16].”
Mobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233 · cited 2×
“In recent years, several studies have demonstrated that it is feasible to achieve sufficient inter-rater reliability with criteria-based rating scales by using trained raters [35,51].”
Enhancing the explanatory power of usability heuristics
Nielsen, J. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191729 · Cited by 1,439
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397 · cited 1×
“Therefore, Usability should be considered to be a barrier rather than a facilitator of user engagement [48,49].”
Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 2×
“Enlight is the first suite of measures to incorporate behavior change, persuasive design, and therapeutic alliance concepts—concepts that have been found to affect a program’s therapeutic potential [15,16].”
Research works printed but not linked: 2
- 10.1111/j.1751-9004.2007.00001 · doi:10.1111/j.1751-9004.2007.00001
- Madan, A, 2012, Int J Eng Sci, 4, 590
Cited by
Objective user engagement with mental health apps: systematic search and panel-based usage analysis
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019) · Journal of Medical Internet Research · doi:10.2196/14567 · Cited by 965
“Relational factors have also been previously acknowledged to nurture a therapeutic alliance with users [45-47], and have demonstrated to be a quality aspect that predicts user engagement with mobile health interventions [28].”
Self-citationBeatty, C., Malik, T., Meheli, S., & Sinha, C. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.847991 · Cited by 224
Tremain, H., McEnery, C., Fletcher, K., & Murray, G. (2020) · JMIR Mental Health · doi:10.2196/17204 · Cited by 193
Baumel, A., Edan, S., & Kane, J. M. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz147 · Cited by 143
Self-citationGamifying CBT to deliver emotional health treatment to young people on smartphones
Christie, G. I., Shepherd, M., Merry, S. N., Hopkins, S., Knightly, S., & Stasiak, K. (2019) · Internet Interventions · doi:10.1016/j.invent.2019.100286 · Cited by 81
Metadata
- Authors
- Baumel, A., & Kane, J. M.
- Affiliations
- University of Haifa (IL)
- Zucker Hillside Hospital (US)
- Corresponding author
- Baumel, Amit
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2018-12-14
- Volume
- 20
- Issue
- 12
- Pages
- e11491
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6315225
- OpenAlex
- W2904531033
- Cited by
- 131
- References
- 50
- Citations by year
- 2026: 12 · 2025: 17 · 2024: 18 · 2023: 25 · 2022: 14 · 2021: 16 · 2020: 18 · 2019: 11
- FWCI
- 17.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- usage data, mobile application, online intervention, therapeutic relationship, product design, usability, visual design, app retention, Google Play reviews, research evidence, credibility, mental health apps
- MeSH terms
- Humans, Persuasive Communication, Self-Help Groups, Telemedicine, Mobile Applications, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
