User Engagement in Mental Health Apps: A Review of Measurement, Reporting, and Validity
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: Despite the potential benefits of mobile mental health apps, real-world results indicate engagement issues because of low uptake and sustained use. This review examined how studies have measured and reported on user engagement indicators (UEIs) for mental health apps. METHODS: A systematic review of multiple databases was performed in July 2018 for studies of mental health apps for depression, bipolar disorder, schizophrenia, and anxiety that reported on UEIs, namely usability, user satisfaction, acceptability, and feasibility. The subjective and objective criteria used to assess UEIs, among other data, were extracted from each study. RESULTS: Of 925 results, 40 studies were eligible. Every study reported positive results for the usability, satisfaction, acceptability, or feasibility of the app. Of the 40 studies, 36 (90%) employed 371 indistinct subjective criteria that were assessed with surveys, interviews, or both, and 23 studies used custom subjective scales, rather than preexisting standardized assessment tools. A total of 25 studies (63%) used objective criteria-with 71 indistinct measures. No two studies used the same combination of subjective or objective criteria to assess UEIs of the app. CONCLUSIONS: The high heterogeneity and use of custom criteria to assess mental health apps in terms of usability, user satisfaction, acceptability, or feasibility present a challenge for understanding real-world low uptake of these apps. Every study reviewed claimed that UEIs for the app were rated highly, which suggests a need for the field to focus on engagement by creating reporting standards and more carefully considering claims.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Software monitoring tools and system logs enabled measuring trial subject UEIs for eHealth products more thoroughly than RCTs for medication trials, but eHealth UEIs were not collected consistently [29].”
From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior chatbot reviews did not compare user engagement indicators (UEI).”
From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While effectiveness is generally assessed through an RCT, evaluation of the feasibility, acceptability, and engagement with dMH interventions requires varied methodologies, and a combination of both subjective and objective criteria are suggested as appropriate for understanding user engagement [20].”
From Feasibility and Acceptability of the Aboriginal and Islander Mental Health Initiative for Youth App: Nonrandomized Pilot With First Nations Young People (2023) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research on technology acceptance among mental health application users shows that, while appropriate design and content can yield high user satisfaction, sustained engagement remains challenging owing to variability in user needs and engagement measurement methods [11].”
From User Experience and Clinical Utility of a Smartphone-Based Application for Hwa-Byung Treatment: a protocol for a single-arm prospective pilot study. (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research on technology acceptance among mental health application users shows that, while appropriate design and content can yield high user satisfaction, sustained engagement remains challenging owing to variability in user needs and engagement measurement methods [11].”
From User Experience and Clinical Utility of a Smartphone-Based Application for Hwa-Byung Treatment: a protocol for a single-arm prospective pilot study. (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies
Venkatesh, V., & Davis, F. D. (2000) · Management Science · doi:10.1287/mnsc.46.2.186.11926 · Cited by 23,066
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Davis, F. D. (1993) · International Journal of Man-Machine Studies · doi:10.1006/imms.1993.1022 · Cited by 2,964
Assessment of client/patient satisfaction: development of a general scale
Larsen, D. L., Attkisson, C. C., Hargreaves, W. A., & Nguyen, T. D. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90094-6 · Cited by 2,887
IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use
Lewis, J. R. (1995) · International Journal of Human-Computer Interaction · doi:10.1080/10447319509526110 · Cited by 2,429
Psychometric properties of the credibility/expectancy questionnaire
Devilly, G. J., & Borkovec, T. D. (2000) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/s0005-7916(00)00012-4 · Cited by 2,295
Firth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097 · cited 1×
“Early evidence concerning the efficacy of mobile mental health apps has created a wave of enthusiasm and support (2).”
BackgroundSelf-citationTorous, J., Nicholas, J., Larsen, M. E., Firth, J., & Christensen, H. (2018) · Evidence Based Mental Health · doi:10.1136/eb-2018-102891 · Cited by 790
Self-citationAssessment of patient satisfaction: Development and refinement of a Service Evaluation Questionnaire
Nguyen, T. D., Attkisson, C. C., & Stegner, B. L. (1983) · Evaluation and Program Planning · doi:10.1016/0149-7189(83)90010-1 · Cited by 736
Harnessing context sensing to develop a mobile intervention for depression
Burns, M. N., Begale, M., Duffecy, J., Gergle, D., Karr, C. J., Giangrande, E., & Mohr, D. C. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1838 · Cited by 646 · cited 2×
“Of these, nine apps were designed for people with depression (11, 12, 19, 21, 27, 29, 40, 43, 46), four with bipolar (16, 20, 22, 47), seven with schizophrenia (23, 25, 28, 33, 38, 39, 45) and seven with anxiety (15, 26, 30, 31, 32, 35, 44).”
Research works printed but not linked: 2
- Dang M, 2016, Stud Health Technol Inform, 228, 195
- Kreyenbuhl J, 2016, Clin Schizophr Relat Psychoses
Cited by
Torous, J., Bucci, S., Bell, I. H., Kessing, L. V., Faurholt‐Jepsen, M., Whelan, P., Carvalho, A. F., Keshavan, M., Linardon, J., & Firth, J. (2021) · World Psychiatry · doi:10.1002/wps.20883 · Cited by 1,199
Self-citationBorghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069
Torous, J., Lipschitz, J., Ng, M., & Firth, J. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.11.167 · Cited by 546
Self-citationGoldberg, S. B., Lam, S. U., Simonsson, O., Torous, J., & Sun, S. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000002 · Cited by 237
“While the first generation of these studies focused on feasibility and acceptability, the accumulating evidence clearly indicates that people suffering from all mental health conditions, including even schizophrenia (which may be associated with severe disability that in theory could interfere with feasibility and acc…”
BackgroundSelf-citationProblems and Barriers Related to the Use of Digital Health Applications: Scoping Review
Giebel, G. D., Speckemeier, C., Abels, C., Plescher, F., Börchers, K., Wasem, J., Blase, N., & Neusser, S. (2023) · Journal of Medical Internet Research · doi:10.2196/43808 · Cited by 229
Saleem, M., Kühne, L., De Santis, K. K., Christianson, L., Brand, T., & Busse, H. (2021) · JMIR Mental Health · doi:10.2196/30000 · Cited by 199
“In addition, there is a need for careful understanding of engagement before claiming engagement strategies used by these apps as effective engagement strategies [43].”
BackgroundBy the numbers: ratings and utilization of behavioral health mobile applications
Carlo, A. D., Hosseini Ghomi, R., Renn, B. N., & Areán, P. A. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0129-6 · Cited by 164
Rethinking (Dis)Engagement in Human-Computer Interaction
O'Brien, H. L., Roll, I., Kampen, A., & Davoudi, N. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107109 · Cited by 139
Digital Health and Engagement—Looking Behind the Measures and Methods
Torous, J., Michalak, E. E., & O’Brien, H. L. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.10918 · Cited by 112
Self-citationChan, A. H. Y., & Honey, M. L. L. (2022) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.12744 · Cited by 112
Digital Mental Health Interventions for Depression: Scoping Review of User Engagement
Lipschitz, J. M., Van Boxtel, R., Torous, J., Firth, J., Lebovitz, J. G., Burdick, K. E., & Hogan, T. P. (2022) · Journal of Medical Internet Research · doi:10.2196/39204 · Cited by 105
“Similarly, user-rated acceptability and feasibility have been evaluated as proxies for engagement [19].”
Self-citationMclaughlin, M., Delaney, T., Hall, A., Byaruhanga, J., Mackie, P., Grady, A., Reilly, K., Campbell, E., Sutherland, R., Wiggers, J., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/23180 · Cited by 105
“Each study used different self-reporting tools to assess different constructs of subjective experience (ie, website usefulness, app likeability, engagement, game-flow, and usability), which has previously been identified as an issue in the assessment of subjective experience engagement [60].”
MethodKaveladze, B. T., Wasil, A. R., Bunyi, J. B., Ramirez, V., & Schueller, S. M. (2022) · JMIR Human Factors · doi:10.2196/30766 · Cited by 104
Leech, T., Dorstyn, D., Taylor, A., & Li, W. (2021) · Children and Youth Services Review · doi:10.1016/j.childyouth.2021.106073 · Cited by 100
Forbes, A., Keleher, M. R., Venditto, M., & DiBiasi, F. (2023) · Journal of Medical Internet Research · doi:10.2196/43727 · Cited by 94
“The search yielded 1181 records, and an additional 590 records were identified by manually searching the reference lists of relevant review articles [57,65-68”
The best person (or machine) for the job: Rethinking task shifting in healthcare
van Schalkwyk, M. C., Bourek, A., Kringos, D. S., Siciliani, L., Barry, M. M., Maeseneer, J. D., & McKee, M. (2020) · Health Policy · doi:10.1016/j.healthpol.2020.08.008 · Cited by 114
Eaton, C. K., Vallejo, N., McDonald, X., Wu, J., Rodríguez, R., Muthusamy, N., Mathioudakis, N. N., & Riekert, K. A. (2024) · Journal of Medical Internet Research · doi:10.2196/50508 · Cited by 140
“…have been further limited by using a very broad definition of engagement to include usability, feasibility, user satisfaction, and acceptability [13], limiting the review to studies in which only postassessment retention data were obtained [11] (excludes interventions earlier in the design…”
Designing m-Health interventions for precision mental health support
Bidargaddi, N., Schrader, G., Klasnja, P., Licinio, J., & Murphy, S. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-00895-2 · Cited by 92
Wasil, A. R., Gillespie, S., Patel, R., Petre, A., Venturo-Conerly, K. E., Shingleton, R. M., Weisz, J. R., & DeRubeis, R. J. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000604 · Cited by 86
Weerdmeester, J., van Rooij, M. M., Engels, R. C., & Granic, I. (2020) · Journal of Medical Internet Research · doi:10.2196/14958 · Cited by 84
Goldberg, S. B., Sun, S., Carlbring, P., & Torous, J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00923-7 · Cited by 75
Self-citationEngagement and attrition in digital mental health: current challenges and potential solutions
Smith, K., Ward, T., Lambe, S., Ostinelli, E. G., Blease, C., Gant, T., Gold, S. M., Holmes, E. A., Paccoud, I., Vinnikova, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01778-w · Cited by 96
“Despite these efforts, there has been huge variability in the terminology applied to both usage and self-report measures 5,6 .”
Self-citationAnmella, G., Sanabra, M., Primé-Tous, M., Segú, X., Cavero, M., Morilla, I., Grande, I., Ruiz, V., Mas, A., Martín-Villalba, I., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/43293 · Cited by 65
“In addition, UEIs still require consistent and replicable standard definitions and valid measures with solid thresholds [98].”
Milne-Ives, M., Homer, S. R., Andrade, J., & Meinert, E. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1227443 · Cited by 69
“, 2020a) and is a potential reason why evidence of their effectiveness remains mixed (Dounavi and Tsoumani, 2019; Ng et al., 2019; Milne-Ives et al., 2020; Moshe et al., 2021).”
Okolo, C. A., Babawarun, O., Arowoogun, J. O., Adeniyi, A. O., & Chidi, R. (2024) · International Journal of Science and Research Archive · doi:10.30574/ijsra.2024.11.1.0334 · Cited by 117
“A study published in the Journal of Medical Internet Research examined the impact of Headspace on stress and anxiety levels in college students (Lipschitz, et. al., 2019, Ng, et. al., 2019, Zhang, et. al., 2019).”
BackgroundJaworski, B. K., Taylor, K., Ramsey, K. M., Heinz, A., Steinmetz, S., Pagano, I., Moraja, G., & Owen, J. E. (2021) · Journal of Medical Internet Research · doi:10.2196/26559 · Cited by 66
“However, as noted by Ng and colleagues [47], there is a need for more standardized reporting of measures related to user engagement and retention.”
BackgroundRating analysis and BERTopic modeling of consumer versus regulated mHealth app reviews in Germany
Uncovska, M., Freitag, B., Meister, S., & Fehring, L. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00862-3 · Cited by 78
Is There an App for That?: Ethical Issues in the Digital Mental Health Response to COVID-19
Skorburg, J. A., & Yam, J. (2022) · AJOB Neuroscience · doi:10.1080/21507740.2021.1918284 · Cited by 64
“Mental illness makes up 13 – 16% of the total global burden of disease, and it was well-known before the pandemic that this disease burden is disproportionately high in low-income areas (Collins et al. 2011; Ngui et al. 2010; Vigo, Thornicroft, and Atun 2016).”
BackgroundJabir, A. I., Martinengo, L., Lin, X., Torous, J., Subramaniam, M., & Tudor Car, L. (2023) · Journal of Medical Internet Research · doi:10.2196/44548 · Cited by 63
“For example, studies that evaluated mental health DHIs typically reported user experience, satisfaction, and engagement with the intervention without clear and standardized criteria to evaluate them [6,8,11].”
Self-citationLimpanopparat, S., Gibson, E., & Harris, D. A. (2024) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2024.100081 · Cited by 62
Bijkerk, L. E., Oenema, A., Geschwind, N., & Spigt, M. (2022) · International Journal of Behavioral Medicine · doi:10.1007/s12529-022-10086-6 · Cited by 93
“[55] found that assess affective engagement are the Client Satisfaction Questionnaire (CSQ; [56]) and the System Usability Scale (SUS; [57]).”
BackgroundBear, H. A., Ayala‐Nunes, L., Ramos, G., Manchanda, T., Fernandes, B., Chabursky, S., Walper, S., Watkins, E. R., & Fazel, M. S. (2024) · Journal of Medical Internet Research · doi:10.2196/48964 · Cited by 78
“This is consistent with the literature that suggests that the majority of those offered these app-based interventions do not engage at the recommended frequency or complete the full course of treatment [58,59].”
Engagement with mobile health interventions for depression: A systematic review
Molloy, A., & Anderson, P. L. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100454 · Cited by 59
Measuring the active elements of cognitive-behavioral therapies
Cohen, Z. D., Barnes-Horowitz, N. M., Forbes, C. N., & Craske, M. G. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104364 · Cited by 58
Kodish, T., Lau, A. S., Gong-Guy, E., Congdon, E. L., Arnaudova, I., Schmidt, M., Shoemaker, L., & Craske, M. G. (2021) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-021-01163-1 · Cited by 69
“1 3 with engagement (Ng et al., 2019), emphasizing the need for streamlined and user-friendly processes to enhance engagement in digital mental health innovations.”
BackgroundO'Brien, H. L., Morton, E., Kampen, A., Barnes, S. J., & Michalak, E. E. (2020) · BJPsych Open · doi:10.1192/bjo.2020.72 · Cited by 57
Hentati, A., Forsell, E., Ljótsson, B., Kaldo, V., Lindefors, N., & Kraepelien, M. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100448 · Cited by 64
“Treatment engagement within DMHS has been conceptualized and measured in different ways (Perski et al., 2017; Ng et al., 2019).”
BackgroundMinen, M. T., Corner, S. E., Berk, T., Levitan, V., Friedman, S., Adhikari, S., & Seng, E. (2021) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2020.12.008 · Cited by 54
Self-citationBear, H. A., Ayala Nunes, L., DeJesus, J., Liverpool, S., Moltrecht, B., Neelakantan, L., Harriss, E., Watkins, E., & Fazel, M. (2022) · Journal of Medical Internet Research · doi:10.2196/40347 · Cited by 51
Haque, M. R., & Rubya, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555146 · Cited by 48
Smartphone Apps for College Mental Health: A Concern for Privacy and Quality of Current Offerings
Melcher, J., & Torous, J. (2020) · Psychiatric Services · doi:10.1176/appi.ps.202000098 · Cited by 50
Self-citationAnmella, G., Faurholt‐Jepsen, M., Hidalgo‐Mazzei, D., Raduà, J., Passos, I. C., Kapczinski, F. P., Minuzzi, L., Alda, M., Meier, S. M., Hájek, T., et al. (2022) · Bipolar Disorders · doi:10.1111/bdi.13243 · Cited by 59
Moylan, K., & Doherty, K. (2025) · Journal of Medical Internet Research · doi:10.2196/67114 · Cited by 46
“Despite this broadly acknowledged situation, 2% of government health spending worldwide is allocated to mental health [2,24,25], a situation that has led to a crisis in access to care.”
Mclaughlin, M., Delaney, T., Hall, A., Byaruhanga, J., Mackie, P., Grady, A., Reilly, K., Campbell, E., Sutherland, R., Wiggers, J., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/29094 · Cited by 45
User Perspectives of Mood-Monitoring Apps Available to Young People: Qualitative Content Analysis
Widnall, E., Grant, C., Wang, T., Cross, L., Velupillai, S. U., Roberts, A., Stewart, R. J., Simonoff, E. A., & Downs, J. M. (2020) · JMIR mhealth and uhealth · doi:10.2196/18140 · Cited by 55
“It is widely acknowledged within the literature that app engagement metrics and reporting remain unstandardized and heterogeneous [18,19].”
Kim, M., Yang, J., Ahn, W.-Y., & Choi, H. J. (2021) · Journal of Medical Internet Research · doi:10.2196/27218 · Cited by 41
“For the first time, this study evaluated the multiple dimensions of motivation at two different periods: before (conventional motivation) and during (in-app motivation) the intervention [31,32].”
BackgroundSmith, K., Hardy, A., Vinnikova, A., Blease, C., Milligan, L., Hidalgo‐Mazzei, D., Lambe, S., Marzano, L., Uhlhaas, P. J., Ostinelli, E. G., et al. (2024) · JMIR Mental Health · doi:10.2196/57155 · Cited by 50
“However, there is also a lack of consensus or standards for assessing and monitoring engagement with and attrition from digital tools, limiting progress in the field [39].”
BackgroundSelf-citationSpottswood, M., Lim, C. T., Davydow, D., & Huang, H. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.892205 · Cited by 40
Ilagan, G. S., Iliakis, E. A., Wilks, C. R., Vahia, I. V., & Choi-Kain, L. W. (2020) · Borderline Personality Disorder and Emotion Dysregulation · doi:10.1186/s40479-020-00127-5 · Cited by 46
Ortiz, A., Maslej, M. M., Husain, M. I., Daskalakis, Z. J., & Mulsant, B. H. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.08.140 · Cited by 48
Bradstreet, S., Allan, S., & Gumley, A. (2019) · Journal of Mental Health · doi:10.1080/09638237.2019.1630727 · Cited by 40
“Ng et al. (2019) proposed that the observed lack of consistency in describing, measuring and reporting engagement could be masking significant problems with usability and safety.”
BackgroundCoaching to Support Mental Health Apps: Exploratory Narrative Review
Meyer, A., Wisniewski, H., & Torous, J. (2022) · JMIR Human Factors · doi:10.2196/28301 · Cited by 34
“A lack of consensus around app engagement measures has also been found in recent reviews [52,53].”
Self-citationLi, S. H., Achilles, M. R., Spanos, S., Habak, S., Werner-Seidler, A., & O’Dea, B. (2022) · The Cognitive Behaviour Therapist · doi:10.1017/s1754470x22000095 · Cited by 39
“To date, engagement in digital mental health interventions has been poorly defined and inconsistently measured (Ng et al., 2019; Perski et al., 2017).”
BackgroundWong, H. W., Lo, B., Shi, J., Hollenberg, E., Abi-Jaoudé, A., Johnson, A., Chaim, G., Cleverley, K., Henderson, J. L., LEVINSON, A. J., et al. (2021) · JMIR Mental Health · doi:10.2196/23447 · Cited by 53
“Unfortunately, many mobile mental health interventions that target this population face low user engagement [8,13,15,16].”
Gumley, A. I., Bradstreet, S., Ainsworth, J. D., Allan, S., Álvarez‐Jiménez, M., Birchwood, M. J., Briggs, A. H., Bucci, S., Cotton, S., Engel, L., et al. (2022) · Health Technology Assessment · doi:10.3310/hlze0479 · Cited by 44
Cevasco, K. E., Morrison Brown, R. E., Woldeselassie, R., & Kaplan, S. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02059-x · Cited by 35
“Software monitoring tools and system logs enabled measuring trial subject UEIs for eHealth prod - ucts more thoroughly than RCTs for medication trials, but eHealth UEIs were not collected consistently [ 29].”
MethodThe conceptualisation and measurement of engagement in digital health
Milne‐Ives, M., Homer, S. R., Andrade, J., & Meinert, E. (2024) · Internet Interventions · doi:10.1016/j.invent.2024.100735 · Cited by 41
“This is made more difficult by the inconsistency in definitions and measures of engagement (Cole-Lewis et al., 2019; Kelders et al., 2020d) and a lack of criteria for what constitutes ‘good ’ engagement (Mclaughlin et al., 2021; Ng et al., 2019).”
BackgroundInfluentialSeegan, P. L., Miller, M. J., Heliste, J. L., Fathi, L., & Mcguire, J. F. (2023) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2023.06.019 · Cited by 48
Toward dynamic phenotypes and the scalable measurement of human behavior
Germine, L. T., Strong, R. W., Singh, S. M., & Sliwinski, M. J. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-020-0757-1 · Cited by 37
“The formidable issue of participant engagement has been welldescribed when it comes to use of digital health apps [14, 15] as well as the limitation sections of many research studies.”
BackgroundWang, L., Fagan, C., & Yu, C.-L. (2020) · Journal of Psychotherapy Integration · doi:10.1037/int0000204 · Cited by 38
Lukka, L., & Palva, J. M. (2023) · JMIR Serious Games · doi:10.2196/42173 · Cited by 46
“” The 2 sides represent complementary aspects of engagement, objective and subjective aspects, which require different approaches in their evaluation [73,74].”
BackgroundCurrey, D., & Torous, J. B. (2023) · Journal of Medical Internet Research · doi:10.2196/39258 · Cited by 42
“Yet, the impact of these apps to date has been minimal, not because apps are ineffective but rather because engagement is often low [3,4].”
Self-citationOpie, J. E., Vuong, A. T., Welsh, E. T., Esler, T. B., Khan, U. R., & Khalil, H. (2024) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-024-00468-5 · Cited by 46
“To date, there are few standardized tools for measuring and evaluating a user’s experience of a digital intervention with reviews reporting heterogeneity in employed measures (Ng et al., 2019; Saleem et al., 2021; Shim et al., 2017; Welsh et al., 2023).”
MethodHardy, A., Ward, T., Emsley, R., Greenwood, K., Freeman, D. H., Fowler, D. G., Kuipers, E., Bebbington, P., & Garety, P. A. (2022) · JMIR Human Factors · doi:10.2196/29725 · Cited by 33
Hugh‐Jones, S., Pert, K., Kendal, S., Eltringham, S., Skelton, C., Yaziji, N., & West, R. (2022) · Mental Health & Prevention · doi:10.1016/j.mhp.2022.200241 · Cited by 42
Patients’ adherence to smartphone apps in the management of bipolar disorder: a systematic review
Patoz, M.-C., Hidalgo-Mazzei, D., Pereira, B., Blanc, O., de Chazeron, I., Murru, A., Verdolini, N., Pacchiarotti, I., Vieta, E., Llorca, P.-M., et al. (2021) · International Journal of Bipolar Disorders · doi:10.1186/s40345-021-00224-6 · Cited by 30
The impact of mindfulness apps on psychological processes of change: a systematic review
Macrynikola, N., Mir, Z., Gopal, T., Rodriguez, E., Li, S., Cox, M., Yeh, G. Y., & Torous, J. B. (2024) · npj Mental Health Research · doi:10.1038/s44184-023-00048-5 · Cited by 42
“The lack of consensus on engagement metrics is a recognized challenge in the mHealth space 79,80 , as is the dif fi culty sustaining engagement over time 81 .”
Self-citationPalmer, C. E., Marshall, E., Millgate, E., Warren, G., Ewbank, M., Cooper, E., Lawes, S., Bouazzaoui, M., Smith, A., Hutchins-Joss, C., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/69351 · Cited by 47
Improving Acceptability and Uptake Behavior for Internet-Based Cognitive-Behavioral Therapy
Molloy, A., Ellis, D. M., Su, L., & Anderson, P. L. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.653686 · Cited by 29
“Overlapping constructs like satisfaction, feasibility, and usability are used interchangeably with acceptability (13).”
BackgroundBell, L., Garnett, C. V., Qian, T., Perski, O., POTTS, H. W. W., & Williamson, E. (2020) · JMIR Research Protocols · doi:10.2196/18690 · Cited by 35
“However, a prime challenge for many behavior change apps is poor levels of engagement, with the frequency (number of sessions) and amount (time spent per session) sharply declining over time for the majority of users [16-18].”
BackgroundUser engagement in mobile apps for people with schizophrenia: A scoping review
de Almeida, R. S., & Marques, A. J. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.1023592 · Cited by 35
“and hinders understanding of what makes apps engaging for different users (18).”
BackgroundInfluentialBrowne, J., Halverson, T. F., & Vilardaga, R. (2021) · Translational Behavioral Medicine · doi:10.1093/tbm/ibab100 · Cited by 29
Bantjes, J. R. (2022) · PLoS Medicine · doi:10.1371/journal.pmed.1004008 · Cited by 27
“One of the biggest obstacles to expanding adolescents’ use of apps and integrating apps into existing services are the poor adherence rates typically observed [10].”
BackgroundComputer-Assisted CBT and Mobile Apps for Depression: Assessment and Integration Into Clinical Care
Wright, J. H., & Mishkind, M. C. (2020) · FOCUS The Journal of Lifelong Learning in Psychiatry · doi:10.1176/appi.focus.20190044 · Cited by 36
Melia, R., Monahan, L., Duggan, J., Bogue, J., O’Sullivan, M., Young, K. L., Chambers, D., & McInerney, S. J. (2021) · BMC Psychiatry · doi:10.1186/s12888-021-03426-5 · Cited by 32
“Despite the reported interest from both patients and professionals in utilizing mobile apps [8, 20], poor uptake of such technologies in clinical settings indicates that m-Health resources suffer from low engagement [21].”
BackgroundGreenwood, K., Gurnani, M., Ward, T., Vogel, E., Vella, C., McGourty, A., Robertson, S., Sacadura, C., Hardy, A., Rus‐Calafell, M., et al. (2022) · Psychology and Psychotherapy Theory Research and Practice · doi:10.1111/papt.12393 · Cited by 24
de Angel, V., Adeleye, F., Zhang, Y., Cummins, N., Munir, S., Lewis, S., Laporta, E., Matcham, F., Sun, S., Folarin, A., et al. (2022) · JMIR Mental Health · doi:10.2196/42866 · Cited by 28
“Work on data availability has generally focused on active approaches [16-18], leaving passive sensing underresearched and underreported [12].”
Value of Engagement in Digital Health Technology Research: Evidence Across 6 Unique Cohort Studies
Goodday, S., Karlin, E., Brooks, A., Chapman, C., Harry, C., Lugo, N., Peabody, S., Rangwala, S., Swanson, E., Tempero, J., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/57827 · Cited by 25
“Definitions of adherence in digital health research studies are heterogeneous [3-6].”
Raffaelli, B., Mecklenburg, J., Overeem, L. H., Scholler, S., Dahlem, M. A., Kurth, T., Gonçalves, A. S. O., Reuter, U., & Neeb, L. (2021) · JMIR mhealth and uhealth · doi:10.2196/26401 · Cited by 27
“The rapid development of modern technologies has led to major advances in health care [1,2].”
BackgroundLin, D. T. S., Nazreen, T., Rutowski, T., Lü, Y., Harati, A., Shriberg, E. E., Chlebek, P., & Aratow, M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.811517 · Cited by 29
“Low engagement rates within mental health apps are a common feasibility challenge (Ng et al., 2019).”
BackgroundMoghimi, E., Knyahnytska, Y., Omrani, M., Nikjoo, N., Stephenson, C., Layzell, G., Simpson, A. I. F., & Alavi, N. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.921527 · Cited by 29
Viduani, A. C., Cosenza, V., Fisher, H. L., Buchweitz, C., Piccin, J., Pereira, R. B., Kohrt, B. A., Mondelli, V., van Heerden, A. C., Araújo, R., et al. (2023) · JMIR Human Factors · doi:10.2196/44388 · Cited by 30
“On the basis of a review of the literature, the following usability indicators [23] were evaluated to define successful implementation [24,25]: (1) acceptance (ie, the proportion of participants who were invited to take part in the IDEABot data collection and agreed to use the tool); (2) initial attrition (ie, failure…”
BackgroundFatouros, P., Tsirmpas, C., Andrikopoulos, D., Kaplow, S., Kontoangelos, K. A., & Papageorgiou, C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01511-7 · Cited by 33
“Layering digital innovation along with the core principles of CBT has proven to be effective 26,27 , however, engagement and retention with DHIs can be challenging 28,29 .”
Mata-Greve, F., Johnson, M., Pullmann, M. D., Friedman, E. C., Fillipo, I. G., Comtois, K. A., & Areán, P. A. (2021) · JMIR Mental Health · doi:10.2196/28360 · Cited by 20
Morton, E. E., Nicholas, J., Lapadat, L., O'Brien, H. L., Barnes, S. J., Poh, C., & Michalak, E. E. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.08.132 · Cited by 28
A systematic evaluation of primary headache management apps leveraging behavior change techniques
Noser, A. E., Klages, K. L., Gamwell, K. L., Brammer, C. N., Hommel, K. A., & Ramsey, R. R. (2021) · Cephalalgia · doi:10.1177/03331024211053572 · Cited by 25
“Fourth, while our use of the MARS engagement subscale provides the most rigorous professional assessment of app engagement currently available (21), recent research highlights a need for more rigorous reporting standards regarding indicators of app user engagement (37).”
MethodMeaningful engagement: A crossfunctional framework for digital therapeutics
Gabe, Flannery, J., Vierra, E., Koepsell, X., Berglund, E., Miller, I., & Lake, J. I. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.890081 · Cited by 24
“Consequently, there is a growing call to follow a clinically informed and data-driven approach to identify specific engagement metrics that uniquely predict the long-term value for a digital therapeutic, referred to as measures of meaningful engagement (4, 9, 11, 12).”
BackgroundDingwall, K. M., Povey, J., Sweet, M., Friel, J., Shand, F., Titov, N., Wormer, J., Mirza, T., & Nagel, T. (2023) · JMIR Human Factors · doi:10.2196/40111 · Cited by 21
“…effectiveness is generally assessed through an RCT, evaluation of the feasibility, acceptability, and engagement with dMH interventions requires varied methodologies, and a combination of both subjective and objective criteria are suggested as appropriate for understanding user engagement [20].”
Aizenstros, A., Bakker, D., Hofmann, S. G., Curtiss, J. E., & Kazantzis, N. (2020) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465820000922 · Cited by 22
“Additionally, inconsistent assessment methods render comparisons between studies difficult (Ng et al., 2019).”
BackgroundLi, S., Achilles, M. R., Werner‐Seidler, A., Beames, J. R., Subotic-Kerry, M., & O’Dea, B. (2022) · JMIR Mental Health · doi:10.2196/37640 · Cited by 24
“Inadequate reporting of instructions for use and significant heterogeneity in reporting of adherence are further demonstrated in at least 5 systematic and narrative reviews of adherence to digital psychological interventions in adults [32-36].”
BackgroundGarety, P. A., Ward, T., Emsley, R., Greenwood, K., Freeman, D. H., Fowler, D. G., Kuipers, E., Bebbington, P., Dunn, G., & Hardy, A. (2021) · Efficacy and Mechanism Evaluation · doi:10.3310/eme08110 · Cited by 23
Nardi, W. R., Roy, A., DUNSIGER, S. I., & Brewer, J. A. (2022) · Journal of Medical Internet Research · doi:10.2196/33696 · Cited by 22
“Summed user engagement metrics, such as those used in these analyses, offer important insights but provide a limited view when considering the practical reality of engagement as a dynamic interaction between the participant and the app over time [22,23,47,57].”
BackgroundMethodA systematic review of smartphone applications for cancer survivors
Changrani, K., Chima, S., Sharma, A. K., Han, G.-G., Sharma, A., McNamara, M., Jefford, M., Emery, J. D., & Druce, P. (2023) · Journal of Cancer Survivorship · doi:10.1007/s11764-023-01435-9 · Cited by 23
“Previous studies have shown that app use decreases over time in this population, and we have a limited understanding how long survivors would continue to use these apps in a real-word situation [57, 58].”
Xie, E. B., Freeman, M., Penner‐Goeke, L., Reynolds, K., Lebel, C. A., Giesbrecht, G. F., Rioux, C., MacKinnon, A., Sauer-Zavala, S. E., Roos, L. E., et al. (2023) · Pilot and Feasibility Studies · doi:10.1186/s40814-023-01245-x · Cited by 32
“Although this is consistent with previous work showing high user satisfaction in mental health app-based interventions, review findings have shown that many previous evaluations of mobile mental health apps have used non-standardized measures of usability and satisfaction, potentially inflating these reported ratings…”
ResultSon, C., Hegde, S., Markert, C., Zahed, K., & Sasangohar, F. (2023) · JMIR Formative Research · doi:10.2196/41018 · Cited by 20
“A systematic literature review of user engagement in mental health apps [44] identified the low sustained use of mobile mental health apps and lack of usability testing standards as persistent challenges.”
Background(2023) · doi:10.2196/47350 · Cited by 18
“%) have demonstrated clinical efficacy [49-51].”
Allan, S., McLeod, H. J., Bradstreet, S., Beedie, S. A., Moir, B., Gleeson, J., Farhall, J., Morton, E. E., & Gumley, A. I. (2019) · JMIR Research Protocols · doi:10.2196/15634 · Cited by 19
“Engagement with digital interventions consists of 2 definitions: first, the extent to which an intervention is actually used (indicated by nonsubjective quantitative measures such as passively recording frequency of intervention usage), and second, as a subjective experience characterized by attention, interest, and a…”
E-mental health applications for depression: an evidence-based ethical analysis
Rubeis, G. (2021) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-019-01093-y · Cited by 17
García-Estela, A., Cantillo, J., Angarita‐Osorio, N., Mur-Milà, E., Anmella, G., Pérez, V., Vieta, E., Hidalgo‐Mazzei, D., & Colom, F. (2022) · Journal of Medical Internet Research · doi:10.2196/31565 · Cited by 24
“However, there is a general tendency among mental health apps toward low retention and engagement rates [32,47].”
Prochaska, J. J., Wang, Y., Bowdring, M. A., Chieng, A., Chaudhary, N. P., & Ramo, D. E. (2023) · JMIR mhealth and uhealth · doi:10.2196/47183 · Cited by 24
“Heterogeneity in engagement metrics has been reported in previous studies on mental health apps [47].”
Result
Metadata
- Authors
- Ng, M. M., Firth, J., Minen, M., & Torous, J.
- Affiliations
- Beth Israel Deaconess Medical Center (US)
- Venue
- Psychiatric Services
- Published
- 2019-07-01
- Volume
- 70
- Issue
- 7
- Pages
- 538-544
- ISSN
- 0022-1597 (ISSN-L)
- 1075-2730
- 1557-9700
- PMC
- PMC6839109
- OpenAlex
- W2924286512
- Cited by
- 293
- References
- 60
- Citations by year
- 2026: 23 · 2025: 60 · 2024: 33 · 2023: 75 · 2022: 40 · 2021: 37 · 2020: 20 · 2019: 5
- FWCI
- 28.69
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Psychiatric Services (publishedVersion)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (K23MH116130)
- National Center for Complementary and Integrative Health (K23AT009706)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- mental health apps, user engagement, engagement metrics, usability, user satisfaction, acceptability, feasibility, measurement validity, reporting guidelines, mobile mental health, depression, bipolar disorder, schizophrenia, anxiety, app uptake
- MeSH terms
- Mental Disorders, Evaluation Studies as Topic, Humans, Outcome and Process Assessment, Health Care, Patient Acceptance of Health Care, Patient Participation, Patient Satisfaction, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.