Clinically Meaningful Use of Mental Health Apps and its Effects on Depression: Mixed Methods Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Background: User engagement is key to the effectiveness of digital mental health interventions. Considerable research has examined the clinical outcomes of overall engagement with mental health apps (eg, frequency and duration of app use). However, few studies have examined how specific app use behaviors can drive change in outcomes. Understanding the clinical outcomes of more nuanced app use could inform the design of mental health apps that are more clinically effective to users. Objective: This study aimed to classify user behaviors in a suite of mental health apps and examine how different types of app use are related to depression and anxiety outcomes. We also compare the clinical outcomes of specific types of app use with those of generic app use (ie, intensity and duration of app use) to understand what aspects of app use may drive symptom improvement. Methods: We conducted a secondary analysis of system use data from an 8-week randomized trial of a suite of 13 mental health apps. We categorized app use behaviors through a mixed methods analysis combining qualitative content analysis and principal component analysis. Regression analyses were used to assess the association between app use and levels of depression and anxiety at the end of treatment. Results: A total of 3 distinct clusters of app use behaviors were identified: learning, goal setting, and self-tracking. Each specific behavior had varied effects on outcomes. Participants who engaged in self-tracking experienced reduced depression symptoms, and those who engaged with learning and goal setting at a moderate level (ie, not too much or not too little) also had an improvement in depression. Notably, the combination of these 3 types of behaviors, what we termed “clinically meaningful use,” accounted for roughly the same amount of variance as explained by the overall intensity of app use (ie, total number of app use sessions). This suggests that our categorization of app use behaviors succeeded in capturing app use associated with better outcomes. However, anxiety outcomes were neither associated with specific behaviors nor generic app use. Conclusions: This study presents the first granular examination of user interactions with mental health apps and their effects on mental health outcomes. It has important implications for the design of mobile health interventions that aim to achieve greater user engagement and improved clinical efficacy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Overall, mHealth approaches to perinatal mental health care allow patients to access and review information about PMADs and may serve to decrease stigma and improve likelihood that patients will seek out treatment [32].”
From Technology-Based Approaches for Supporting Perinatal Mental Health (2022) · cites this paper as [32]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910
A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7
Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.10.1092 · Cited by 31,765
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 1×
“Mental health apps can enhance skill building, deliver psychoeducation, and facilitate self-monitoring, thereby reducing symptoms of depression and anxiety [2,5,6].”
Persuasive Systems Design: Key Issues, Process Model, and System Features
Oinas-Kukkonen, H., & Harjumaa, M. (2009) · Communications of the Association for Information Systems · doi:10.17705/1cais.02428 · Cited by 1,663
Perski, O., Blandford, A., West, R., & Michie, S. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0453-1 · Cited by 1,193
Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions
Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., Merchant, G. C., Naughton, F., & Blandford, A. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.015 · Cited by 1,118
A stage-based model of personal informatics systems
Li, I., Dey, A., & Forlizzi, J. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753409 · Cited by 1,100
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1602 · Cited by 1,097 · cited 1×
“Coaching was based on a low-intensity coaching model [29] that relied primarily on brief SMS text messaging (2-3 messages per week) to promote engagement.”
Self-citationTorous, J., Nicholas, J., Larsen, M. E., Firth, J., & Christensen, H. (2018) · Evidence Based Mental Health · doi:10.1136/eb-2018-102891 · Cited by 790
Self-citationThe development and evaluation of a survey to measure user engagement
O'Brien, H. L., & Toms, E. G. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21229 · Cited by 790
Alone Together: Why We Expect More from Technology and Less from Each Other
Cheung, J. C.-S. (2013) · Journal of Social Work Practice · doi:10.1080/02650533.2013.769209 · Cited by 743
Firth, J., Torous, J., Nicholas, J., Carney, R., Rosenbaum, S., & Sarris, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.04.046 · Cited by 675 · cited 1×
“Mental health apps can enhance skill building, deliver psychoeducation, and facilitate self-monitoring, thereby reducing symptoms of depression and anxiety [2,5,6].”
Self-citationInteraction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI
Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055101 · Cited by 576 · cited 1×
“This finding corresponds to prior work suggesting that people tend to use mobile apps in very short bursts of time, given their habit of using smartphones in spare moments [39,40].”
Mobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438 · cited 1×
“These mobile technologies incorporate a wide array of system features and strategies that foster user engagement and promote behavior change, such as customization, reminders, self-monitoring, rewards, and peer support [3,7-10].”
Self-citationNeff, G., & Nafus, D. (2016) · The MIT Press eBooks · doi:10.7551/mitpress/10421.001.0001 · Cited by 432
Short, C. E., DeSmet, A., Woods, C., Williams, S. L., Maher, C., Middelweerd, A., Müller, A. M., Wark, P. A., Vandelanotte, C., Poppe, L., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9397 · Cited by 400 · cited 2×
“Engagement has been inconsistently viewed as a multidimensional construct, variably encompassing behavioral, emotional, and cognitive factors [14,15].”
Exploring Goal-setting, Rewards, Self-monitoring, and Sharing to Motivate Physical Activity
Munson, S., & Consolvo, S. (2012) · Proceedings of the 6th International Conference on Pervasive Computing Technologies for Healthcare · doi:10.4108/icst.pervasivehealth.2012.248691 · Cited by 355
Huckvale, K., Torous, J., & Larsen, M. E. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.2542 · Cited by 312 · cited 1×
“…in relation to mental health apps was highlighted in 2 recent studies, which suggest that currently available mental health apps often misuse user data [42] and that users’ willingness to share personal sensing data varies depending on the type of data collected and with whom they are shared [43].”
Designing mobile health technology for bipolar disorder: a field trial of the monarca system
Bardram, J. E., Frost, M., Szántó, K., Faurholt-Jepsen, M., Vinberg, M., & Kessing, L. V. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481364 · Cited by 304
IntelliCare: An Eclectic, Skills-Based App Suite for the Treatment of Depression and Anxiety
Mohr, D. C., Tomasino, K. N., Lattie, E. G., Palac, H. L., Kwasny, M. J., Weingardt, K., Karr, C. J., Kaiser, S. M., Rossom, R. C., Bardsley, L. R., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6645 · Cited by 301
Self-citationTorous, J., Friedman, R., & Keshavan, M. (2014) · JMIR mHealth and uHealth · doi:10.2196/mhealth.2994 · Cited by 287 · cited 1×
“Given the accessibility and ubiquity of mobile phones, researchers and clinicians have increasingly leveraged mobile phone apps to deliver health interventions and enhance self-management of chronic conditions such as depression and anxiety [1-4].”
Engagement in HCI: Conception, Theory and Measurement
Doherty, K., & Doherty, G. (2019) · ACM Computing Surveys · doi:10.1145/3234149 · Cited by 264
Engagement and retention: measuring breadth and depth of participant use of an online intervention
Couper, M. P., Alexander, G. L., Zhang, N., Little, R. J., Maddy, N., Nowak, M. A., McClure, J. B., Calvi, J. J., Rolnick, S. J., Stopponi, M. A., et al. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1430 · Cited by 244
Engagement in a Diabetes Self-management Website: Usage Patterns and Generalizability of Program Use
Glasgow, R. E., Christiansen, S. M., Kurz, D., King, D. K., Woolley, T., Faber, A. J., Estabrooks, P. A., Strycker, L., Toobert, D., & Dickman, J. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1391 · Cited by 219 · cited 1×
“…with the app in terms of intensity (eg, number of app use sessions [19]), frequency (eg, number/percentage of days the app is used [20]), duration (eg, time spent on the app [21,22]), time between first and last app use [19,20]), and types (eg, passive, active, reflective, and didactic [23,24]).”
Doherty, G., Coyle, D., & Sharry, J. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208602 · Cited by 198 · cited 1×
“These mobile technologies incorporate a wide array of system features and strategies that foster user engagement and promote behavior change, such as customization, reminders, self-monitoring, rewards, and peer support [3,7-10].”
Stawarz, K., Preist, C., Tallon, D., Wiles, N., & Coyle, D. (2018) · Journal of Medical Internet Research · doi:10.2196/10120 · Cited by 198
Mobile phones as medical devices in mental disorder treatment: an overview
Gravenhorst, F., Muaremi, A., Bardram, J., Grünerbl, A., Mayora, O., Wurzer, G., Frost, M., Osmani, V., Arnrich, B., Lukowicz, P., et al. (2015) · Personal and Ubiquitous Computing · doi:10.1007/s00779-014-0829-5 · Cited by 197
Torous, J. B., Chan, S. R., Gipson, S. Y.-M. T., Kim, J. W., Nguyen, T.-Q., Luo, J., & Wang, P. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700423 · Cited by 184 · cited 2×
“Within the wider context, research indicates that app design and quality assessment must also consider users’lived experience, app usability and stability, and data privacy and security [41].”
User engagement in online News: Under the scope of sentiment, interest, affect, and gaze
Arapakis, I., Lalmas, M., Cambazoglu, B. B., Marcos, M., & Jose, J. M. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23096 · Cited by 131
Pham, Q., Graham, G., Carrion, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A. (2019) · JMIR mHealth and uHealth · doi:10.2196/11941 · Cited by 131 · cited 1×
“In this study, we define user engagement as a behavioral experience that involves people’s physical interaction with mobile phone apps [18].”
Mohr, D. C., Schueller, S. M., Tomasino, K. N., Kaiser, S. M., Alam, N., Karr, C., Vergara, J. L., Gray, E. L., Kwasny, M. J., & Lattie, E. G. (2019) · Journal of Medical Internet Research · doi:10.2196/13609 · Cited by 116 · cited 2×
“This is consistent with the findings in the main trial, where significant reductions in anxiety symptom were not related to number of app sessions or time between first and last app use but were only associated with the number of app downloads [27].”
Self-citationMaintenance of weight loss in overweight middle-aged women through the Internet
Cussler, E. C., Teixeira, P. J., Going, S. B., Houtkooper, L. B., Metcalfe, L. L., Blew, R. M., Ricketts, J. R., Lohman, J., Stanford, V. A., & Lohman, T. G. (2008) · Obesity · doi:10.1038/oby.2008.19 · Cited by 115 · cited 1×
“These data quantitatively capture how users physically interact with the app in terms of intensity (eg, number of app use sessions [19]), frequency (eg, number/percentage of days the app is used [20]), duration (eg, time spent on the app [21,22]), time between first and last app use [19,20]), and types (eg, passive, a…”
BackgroundUptake and usage of IntelliCare: A publicly available suite of mental health and well-being apps
Lattie, E. G., Schueller, S. M., Sargent, E., Stiles-Shields, C., Tomasino, K. N., Corden, M. E., Begale, M., Karr, C. J., & Mohr, D. C. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.003 · Cited by 113 · cited 2×
“These data quantitatively capture how users physically interact with the app in terms of intensity (eg, number of app use sessions [19]), frequency (eg, number/percentage of days the app is used [20]), duration (eg, time spent on the app [21,22]), time between first and last app use [19,20]), and types (eg, passive, a…”
BackgroundSelf-citationTwitch crowdsourcing: crowd contributions in short bursts of time
Vaish, R., Wyngarden, K., Chen, J., Cheung, B., & Bernstein, M. S. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2556996 · Cited by 95 · cited 2×
“Although duration of engagement plays a critical role in Web-based interventions [21], the current ways in which people interact with smartphones suggest that mHealth apps should be designed to be quick to use, have simple interactions, and support a single or limited set of related tasks [2,40].”
Multimodal e-mental health treatment for depression: a feasibility trial
Mohr, D. C., Duffecy, J., Jin, L., Ludman, E. J., Lewis, A., Begale, M., & McCarthy Jr, M. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1370 · Cited by 82
Self-citationNicholas, J., Shilton, K., Schueller, S. M., Gray, E. L., Kwasny, M. J., & Mohr, D. C. (2019) · JMIR mHealth and uHealth · doi:10.2196/12578 · Cited by 77 · cited 1×
“…in relation to mental health apps was highlighted in 2 recent studies, which suggest that currently available mental health apps often misuse user data [42] and that users’ willingness to share personal sensing data varies depending on the type of data collected and with whom they are shared [43].”
Self-citationDavies, C., Corry, K., Itallie, A. K. V., Vandelanotte, C., Caperchione, C. M., & Mummery, W. K. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.1792 · Cited by 68
Mattila, E., Lappalainen, R., Välkkynen, P., Sairanen, E., Lappalainen, P., Karhunen, L., Peuhkuri, K., Korpela, R., Kolehmainen, M., & Ermes, M. (2016) · JMIR mhealth and uhealth · doi:10.2196/mhealth.5241 · Cited by 67
Cheung, K., Ling, W., Karr, C. J., Weingardt, K., Schueller, S. M., & Mohr, D. C. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy023 · Cited by 61
Self-citationGraham, M. L., Strawderman, M. S., Demment, M., & Olson, C. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6644 · Cited by 39
Kwasny, M. J., Schueller, S. M., Lattie, E., Gray, E. L., & Mohr, D. C. (2019) · JMIR Mental Health · doi:10.2196/11572 · Cited by 37
Self-citation
Research works printed but not linked: 1
- Lupton, D, 2016, The Quantified Self
Cited by
Challenges in Participant Engagement and Retention Using Mobile Health Apps: Literature Review
Amagai, S., Pila, S., Kaat, A. J., Nowinski, C. J., & Gershon, R. C. (2022) · Journal of Medical Internet Research · doi:10.2196/35120 · Cited by 343
“Refer to Figure 1 for the study selection flowchart and Multimedia Appendix 1 [4-8,17-73] for characteristics of the studies.”
BackgroundSaleem, M., Kühne, L., De Santis, K. K., Christianson, L., Brand, T., & Busse, H. (2021) · JMIR Mental Health · doi:10.2196/30000 · Cited by 199
“” The findings of Zhang et al [47] suggest that digital apps addressing mental health should follow the Goldilocks concept of just right.”
BackgroundInfluentialGraham, A. K., Greene, C. J., Kwasny, M. J., Kaiser, S. M., Lieponis, P., Powell, T., & Mohr, D. C. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2020.1011 · Cited by 162
Self-citationLutz, J., Offidani, E., Taraboanta, L., Lakhan, S. E., & Campellone, T. R. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.823977 · Cited by 89
Technology-Based Approaches for Supporting Perinatal Mental Health
Novick, A. M., Kwitowski, M., Dempsey, J., Cooke, D. L., & Dempsey, A. G. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01349-w · Cited by 35
“Overall, mHealth approaches to perinatal mental health care allow patients to access and review information about PMADs and may serve to decrease stigma and improve likelihood that patients will seek out treatment [32].”
Background
Metadata
- Authors
- Zhang, R., Nicholas, J., Knapp, A. A., Graham, A. K., Gray, E., Kwasny, M. J., Reddy, M., & Mohr, D. C.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Zhang, Renwen
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-12-20
- Volume
- 21
- Issue
- 12
- Pages
- e15644
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6942194
- OpenAlex
- W2980344427
- Cited by
- 91
- References
- 42
- Citations by year
- 2026: 7 · 2025: 11 · 2024: 14 · 2023: 11 · 2022: 19 · 2021: 22 · 2020: 7
- FWCI
- 13.50
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (UL1TR001422)
- National Institute of Mental Health (R01MH100482, T32MH115882)
- National Institute of Diabetes and Digestive and Kidney Diseases (K01DK116925)
- National Center for Advancing Translational Sciences
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
- Keywords
- mental health apps, depression, anxiety, user engagement, self-tracking, goal setting, learning, mobile health intervention, digital mental health, symptom improvement
- MeSH terms
- Adolescent, Adult, Aged, Depressive Disorder, Humans, Mental Health, Middle Aged, Telemedicine, Young Adult, Meaningful Use, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

