Evaluation of a recommender app for apps for the treatment of depression and anxiety: an analysis of longitudinal user engagement
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Objective: While depression and anxiety are common mental health issues, only a small segment of the population has access to standard one-on-one treatment. The use of smartphone apps can fill this gap. An app recommender system may help improve user engagement of these apps and eventually symptoms. Methods: IntelliCare was a suite of apps for depression and anxiety, with a Hub app that provided app recommendations aiming to increase user engagement. This study captured the records of 8057 users of 12 apps. We measured overall engagement and app-specific usage longitudinally by the number of weekly app sessions (“loyalty”) and the number of days with app usage (“regularity”) over 16 weeks. Hub and non-Hub users were compared using zero-inflated Poisson regression for loyalty, linear regression for regularity, and Cox regression for engagement duration. Adjusted analyses were performed in 4561 users for whom we had baseline characteristics. Impact of Hub recommendations was assessed using the same approach. Results: When compared to non-Hub users in adjusted analyses, Hub users had a lower risk of discontinuing IntelliCare (hazard ratio = 0.67, 95% CI, 0.62-0.71), higher loyalty (2- to 5-fold), and higher regularity (0.1–0.4 day/week greater). Among Hub users, Hub recommendations increased app-specific loyalty and regularity in all 12 apps. Discussion/Conclusion: Centralized app recommendations increase overall user engagement of the apps, as well as app-specific usage. Further studies relating app usage to symptoms can validate that such a recommender improves clinical benefits and does so at scale.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rebooting psychotherapy research and practice to reduce the burden of mental illness
Kazdin, A. E., & Blase, S. L. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393527 · Cited by 1,254
Firth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097
Firth, J., Torous, J., Nicholas, J., Carney, R., Rosenbaum, S., & Sarris, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.04.046 · Cited by 675
Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI
Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055101 · Cited by 576
How to evaluate technologies for health behavior change in HCI research
Klasnja, P., Consolvo, S., & Pratt, W. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979396 · Cited by 414
Beyond self-report: tools to compare estimated and real-world smartphone use
Andrews, S., Ellis, D. A., Shaw, H., & Piwek, L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0139004 · Cited by 408
Torous, J., & Roberts, L. W. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0262 · Cited by 362
Perceived barriers to psychological treatments and their relationship to depression
Mohr, D. C., Ho, J., Duffecy, J., Baron, K. G., Lehman, K. A., Jin, L., & Reifler, D. (2010) · Journal of Clinical Psychology · doi:10.1002/jclp.20659 · Cited by 347
Self-citationIntelliCare: An Eclectic, Skills-Based App Suite for the Treatment of Depression and Anxiety
Mohr, D. C., Tomasino, K. N., Lattie, E. G., Palac, H. L., Kwasny, M. J., Weingardt, K., Karr, C. J., Kaiser, S. M., Rossom, R. C., Bardsley, L. R., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6645 · Cited by 301
Self-citationBarriers to psychotherapy among depressed and nondepressed primary care patients
Mohr, D. C., Hart, S. L., Howard, I., Julian, L., Vella, L., Catledge, C., & Feldman, M. D. (2006) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3203_12 · Cited by 276
Self-citationKlasnja, P., Smith, S., Seewald, N. J., Lee, A., Hall, K., Luers, B., Hekler, E. B., & Murphy, S. A. (2019) · Annals of Behavioral Medicine · doi:10.1093/abm/kay067 · Cited by 241
Van Ameringen, M., Turna, J., Khalesi, Z., Pullia, K., & Patterson, B. (2017) · Depression and Anxiety · doi:10.1002/da.22657 · Cited by 234
mHealth in the Wild: Using Novel Data to Examine the Reach, Use, and Impact of PTSD Coach
Owen, J. E., Jaworski, B. K., Kuhn, E., Makin-Byrd, K. N., Ramsey, K. M., & Hoffman, J. E. (2015) · JMIR Mental Health · doi:10.2196/mental.3935 · Cited by 216
Helander, E., Kaipainen, K., Korhonen, I., & Wansink, B. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3084 · Cited by 143
Taki, S., Lymer, S., Russell, C. G., Campbell, K., Laws, R., Ong, K.-L., Elliott, R., & Denney-Wilson, E. (2017) · JMIR mHealth and uHealth · doi:10.2196/mhealth.7236 · Cited by 114
Uptake and usage of IntelliCare: A publicly available suite of mental health and well-being apps
Lattie, E. G., Schueller, S. M., Sargent, E., Stiles-Shields, C., Tomasino, K. N., Corden, M. E., Begale, M., Karr, C. J., & Mohr, D. C. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.003 · Cited by 113
Self-citationRichards, D., Richardson, T., Timulak, L., & McElvaney, J. (2015) · Internet Interventions · doi:10.1016/j.invent.2015.07.003 · Cited by 111
Kelders, S. M., Bohlmeijer, E. T., Pots, W. T. M., & van Gemert-Pijnen, J. E. W. C. (2015) · Behaviour Research and Therapy · doi:10.1016/j.brat.2015.06.014 · Cited by 105
Twitch crowdsourcing: crowd contributions in short bursts of time
Vaish, R., Wyngarden, K., Chen, J., Cheung, B., & Bernstein, M. S. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2556996 · Cited by 95
Cheung, Y. K., Chakraborty, B., & Davidson, K. W. (2015) · Biometrics · doi:10.1111/biom.12258 · Cited by 55
Exploring user learnability and learning performance in an app for depression: usability study
Stiles-Shields, C., Montague, E., Lattie, E. G., Schueller, S. M., Kwasny, M. J., & Mohr, D. C. (2017) · JMIR Human Factors · doi:10.2196/humanfactors.7951 · Cited by 29
Cited by
Objective user engagement with mental health apps: systematic search and panel-based usage analysis
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019) · Journal of Medical Internet Research · doi:10.2196/14567 · Cited by 965
“The retention slope presented a slower decline in app open rates between day 15 and 30 and, based on previous reports, it would be reasonable to assume that there is a continuous usage decline over time (eg, [19,50]), but more studies are needed to determine the magnitude of the decline.”
BackgroundWasil, A. R., Venturo-Conerly, K. E., Shingleton, R. M., & Weisz, J. R. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103498 · Cited by 160
Mohr, D. C., Schueller, S. M., Tomasino, K. N., Kaiser, S. M., Alam, N., Karr, C., Vergara, J. L., Gray, E. L., Kwasny, M. J., & Lattie, E. G. (2019) · Journal of Medical Internet Research · doi:10.2196/13609 · Cited by 116
Self-citationClinically Meaningful Use of Mental Health Apps and its Effects on Depression: Mixed Methods Study
Zhang, R., Nicholas, J., Knapp, A. A., Graham, A. K., Gray, E., Kwasny, M. J., Reddy, M., & Mohr, D. C. (2019) · Journal of Medical Internet Research · doi:10.2196/15644 · Cited by 91
Self-citationWhat Works Best to Engage Participants in Mobile App Interventions and e-Health: A Scoping Review
Oakley-Girvan, I., Yunis, R., Longmire, M., & Ouillon, J. S. (2022) · Telemedicine and e-Health · doi:10.1089/tmj.2021.0176 · Cited by 92
Metadata
- Authors
- Cheung, K., Ling, W., Karr, C. J., Weingardt, K., Schueller, S. M., & Mohr, D. C.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-04-06
- Volume
- 25
- Issue
- 8
- Pages
- 955-962
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2795629330
- Cited by
- 61
- References
- 21
- FWCI
- 8.43
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.