Determination of Markers of Successful Implementation of Mental Health Apps for Young People: Systematic Review
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Background: Smartphone apps have the potential to address some of the current issues facing service provision for young people’s mental health by improving the scalability of evidence-based mental health interventions. However, very few apps have been successfully implemented, and consensus on implementation measurement is lacking. Objective: This review aims to determine the proportion of evidence-based mental health and well-being apps that have been successfully adopted and sustained in real-world settings. A secondary aim is to establish if key implementation determinants such as coproduction, acceptability, feasibility, appropriateness, and engagement contribute toward successful implementation and longevity. Methods: Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, an electronic search of 5 databases in 2021 yielded 18,660 results. After full-text screening, 34 articles met the full eligibility criteria, providing data on 29 smartphone apps studied with individuals aged 15 to 25 years. Results: Of 34 studies, only 10 (29%) studies were identified that were evaluating the effectiveness of 8 existing, commercially available mental health apps, and the remaining 24 (71%) studies reported the development and evaluation of 21 newly developed apps, of which 43% (9/21) were available, commercially or otherwise (eg, in mental health services), at the time of enquiry. Most studies addressed some implementation components including adoption, acceptability, appropriateness, feasibility, and engagement. Factors including high cost, funding constraints, and lengthy research processes impeded implementation. Conclusions: Without addressing common implementation drivers, there is considerable redundancy in the translation of mobile mental health research findings into practice. Studies should embed implementation strategies from the outset of the planned research, build collaborations with partners already working in the field (academic and commercial) to capitalize on existing interventions and platforms, and modify and evaluate them for local contexts or target problems and populations. Trial Registration: PROSPERO CRD42021224365; https://tinyurl.com/4umpn85f
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 1×
“Egilsson et al [43] assessed acceptability with the Systematic Usability Scale, a widely used and relatively well-studied 10-item questionnaire on app usability, where scores range from 0 to 100, and a total score of >70 indicates satisfactory usability and user acceptance [71].”
MethodThe Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737 · cited 1×
“The mixed methods appraisal tool (version 2018) was used to assess the methodological quality of the included studies [33].”
MethodMaking sense of implementation theories, models and frameworks
Nilsen, P. (2015) · Implementation Science · doi:10.1186/s13012-015-0242-0 · Cited by 4,447
Age of onset of mental disorders: a review of recent literature
Kessler, R. C., Amminger, G. P., Aguilar-Gaxiola, S., Alonso, J., Lee, S., & ??st??n, T. B. (2007) · Current Opinion in Psychiatry · doi:10.1097/yco.0b013e32816ebc8c · Cited by 3,457 · cited 1×
“Most common mental health disorders, including depression and anxiety, have their onset during adolescence and, if not successfully resolved, can lead to negative impacts well into adulthood [4,5].”
BackgroundThe answer is 17 years, what is the question: understanding time lags in translational research
Morris, Z. S., Wooding, S., & Grant, J. (2011) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2011.110180 · Cited by 3,340 · cited 1×
“This can be attributed, in part, to the complex and lengthy process of implementing and maintaining evidence-based approaches in practice, as well as the commercial and regulatory complexities of scaling up mobile technologies in health services [12,13].”
BackgroundFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 1×
“Most studies of implemented apps reported them as acceptable, with high user satisfaction and ease of use among young people and health care providers [43,45,61,69].”
BackgroundObjective user engagement with mental health apps: systematic search and panel-based usage analysis
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019) · Journal of Medical Internet Research · doi:10.2196/14567 · Cited by 965 · cited 1×
“As evidence suggests that engagement is the highest in the first days or weeks of downloading an app, it might be more appropriate for apps to be developed with this anticipated behavior in mind [72].”
BackgroundLinardon, J., Cuijpers, P., Carlbring, P., Messer, M., & Fuller‐Tyszkiewicz, M. (2019) · World Psychiatry · doi:10.1002/wps.20673 · Cited by 860
Systematic Review and Meta-Analysis: Adolescent Depression and Long-Term Psychosocial Outcomes
Clayborne, Z. M., Varin, M., & Colman, I. (2019) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2018.07.896 · Cited by 787 · cited 1×
“Most common mental health disorders, including depression and anxiety, have their onset during adolescence and, if not successfully resolved, can lead to negative impacts well into adulthood [4,5].”
BackgroundFleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9275 · Cited by 670 · cited 1×
“Foremost, implementation processes and outcomes from research trials are seldom recorded or reported, and implementation efforts often lack a solid theoretical or model-based approach, making it difficult to understand and explain how and why implementation succeeds or fails [18,19].”
BackgroundHuberty, J., Green, J., Glissmann, C., Larkey, L., Puzia, M., & Lee, C. (2019) · JMIR mHealth and uHealth · doi:10.2196/14273 · Cited by 488
Liverpool, S., Mota, C. P., Sales, C. M. D., Čuš, A., Carletto, S., Hancheva, C., Sousa, S., Cerón, S. C., Moreno-Peral, P., Pietrabissa, G., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/16317 · Cited by 408
Self-citationTorous, J., & Roberts, L. W. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0262 · Cited by 362
Flett, J. A. M., Hayne, H., Riordan, B. C., Thompson, L. M., & Conner, T. S. (2019) · Mindfulness · doi:10.1007/s12671-018-1050-9 · Cited by 338
Practitioner review: Co‐design of digital mental health technologies with children and young people
Bevan Jones, R., Stallard, P., Agha, S. S., Rice, S., Werner‐Seidler, A., Stasiak, K., Kahn, J., Simpson, S. A., Alvarez‐Jimenez, M., Rice, F., et al. (2020) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13258 · Cited by 317
User engagement in mental health apps: A review of measurement, reporting, and validity
Ng, M. M., Firth, J., Minen, M., & Torous, J. (2019) · Psychiatric Services · doi:10.1176/appi.ps.201800519 · Cited by 293
Szinay, D., Jones, A., Chadborn, T., Brown, J., & Naughton, F. (2020) · Journal of Medical Internet Research · doi:10.2196/17572 · Cited by 289 · cited 1×
“Besides innovation and efficacy, other factors, including user engagement, usability, acceptability, accessibility, and low cost, are key prerequisites for adoption, scalability, and uptake [14-17].”
BackgroundSmartphone and tablet self management apps for asthma
Marcano Belisario, J. S., Huckvale, K., Greenfield, G., Car, J., & Gunn, L. H. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010013.pub2 · Cited by 288 · cited 1×
“[25], and the terms for apps were derived from Cochrane reviews [28,29].”
MethodUse of Digital Mental Health for Marginalized and Underserved Populations
Schueller, S. M., Hunter, J. F., Figueroa, C., & Aguilera, A. (2019) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-019-00181-z · Cited by 268
Franklin, J. C., Fox, K. R., Franklin, C. R., Kleiman, E. M., Ribeiro, J. D., Jaroszewski, A. C., Hooley, J. M., & Nock, M. K. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000093 · Cited by 217
Schlosser, D. A., Campellone, T. R., Truong, B., Etter, K., Vergani, S., Komaiko, K., & Vinogradov, S. (2018) · Schizophrenia Bulletin · doi:10.1093/schbul/sby078 · Cited by 206
Actissist: Proof-of-Concept Trial of a Theory-Driven Digital Intervention for Psychosis
Bucci, S., Barrowclough, C., Ainsworth, J., Machin, M., Morris, R., Berry, K., Emsley, R., Lewis, S., Edge, D., Buchan, I., et al. (2018) · Schizophrenia Bulletin · doi:10.1093/schbul/sby032 · Cited by 196 · cited 1×
“The process of applying for funding, app development, data collection, data analysis, and app release is lengthy and expensive, taking up to 6 years [39].”
MethodReid, S. C., Kauer, S. D., Hearps, S. J., Crooke, A. H., Khor, A. S., Sanci, L. A., & Patton, G. C. (2011) · BMC Family Practice · doi:10.1186/1471-2296-12-131 · Cited by 192
Developing mental health mobile apps: Exploring adolescents’ perspectives
Kenny, R., Dooley, B., & Fitzgerald, A. (2016) · Health Informatics Journal · doi:10.1177/1460458214555041 · Cited by 175 · cited 1×
“Besides innovation and efficacy, other factors, including user engagement, usability, acceptability, accessibility, and low cost, are key prerequisites for adoption, scalability, and uptake [14-17].”
BackgroundLau, N., O'Daffer, A., Colt, S., Yi-Frazier, J. P., Palermo, T. M., McCauley, E., & Rosenberg, A. R. (2020) · JMIR mhealth and uhealth · doi:10.2196/17798 · Cited by 174 · cited 1×
“As noted by others, currently, the mental health and well-being apps that have been rigorously evaluated struggle to attract users, and apps with many users are rarely evaluated [76,77].”
BackgroundSchueller, S. M., Neary, M., O'Loughlin, K., & Adkins, E. C. (2018) · Journal of Medical Internet Research · doi:10.2196/10141 · Cited by 167
Hermes, E. D., Lyon, A. R., Schueller, S. M., & Glass, J. E. (2019) · Journal of Medical Internet Research · doi:10.2196/11752 · Cited by 149 · cited 2×
“…of mobile mental health apps, successful implementation can be measured by the extent to which the intervention has been embedded into service provision, the number of app users, the frequency of app use, app engagement, and evidence of sustained use following the end of a research trial [20,21].”
BackgroundEvaluation of an mHealth App (DeStressify) on University Students’ Mental Health: Pilot Trial
Lee, R. A., & Jung, M. E. (2018) · JMIR Mental Health · doi:10.2196/mental.8324 · Cited by 138 · cited 8×
“A consistent and noteworthy finding across several studies was that engagement decreased over time [43,54,67].”
BackgroundResultInfluentialRodgers, R. F., Donovan, E., Cousineau, T., Yates, K., McGowan, K., Cook, E., Lowy, A. S., & Franko, D. L. (2018) · Journal of Youth and Adolescence · doi:10.1007/s10964-017-0804-3 · Cited by 125
Uptake and usage of IntelliCare: A publicly available suite of mental health and well-being apps
Lattie, E. G., Schueller, S. M., Sargent, E., Stiles-Shields, C., Tomasino, K. N., Corden, M. E., Begale, M., Karr, C. J., & Mohr, D. C. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.003 · Cited by 113 · cited 1×
“However, it is recognized that operationalizing meaningful app engagement is not straightforward, as many downloaded apps are never used and more work is needed to define sustained engagement, what leads to it, and how to create products that achieve it [73,74].”
BackgroundSchlosser, D., Campellone, T., Kim, D., Truong, B., Vergani, S., Ward, C., & Vinogradov, S. (2016) · JMIR Research Protocols · doi:10.2196/resprot.5450 · Cited by 103 · cited 1×
“…users) [61,69]; market research and beta testing [54]; design workshops with 15 key stakeholders, followed by a series of in-depth interviews [67,70]; focus groups with young people with lived experience that guided the development of app functionalities [42]; and study groups with teenagers…”
BackgroundPonzo, S., Morelli, D., Kawadler, J. M., Hemmings, N. R., Bird, G., & Plans, D. (2020) · JMIR mHealth and uHealth · doi:10.2196/17767 · Cited by 100
Aschbrenner, K. A., Naslund, J. A., Tomlinson, E. F., Kinney, A., Pratt, S. I., & Brunette, M. F. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00178 · Cited by 99 · cited 1×
“Given the increasing number of young people (between the ages of 15 and 25 years) using digital technologies, smartphone-based interventions provide a scalable solution to support this group to manage their mental health and well-being [6].”
BackgroundFlett, J. A. M., Conner, T. S., Riordan, B. C., Patterson, T., & Hayne, H. (2020) · Psychology & Health · doi:10.1080/08870446.2019.1711089 · Cited by 84 · cited 6×
“%) evaluated an app still available on the market: Headspace [44,46,47,68], Calm [49], Pacifica/Sanvello [38], Smiling Mind [47], Stop Breathe Think [56], and Thrive [59].”
BackgroundMethodInfluentialHur, J.-W., Kim, B., Park, D., & Choi, S.-W. (2018) · Telemedicine and e-Health · doi:10.1089/tmj.2017.0214 · Cited by 82 · cited 1×
“Of those contacted (28/34, 82%), we collected 23 survey responses ( M u l t i m e d i a A p p e n d i x 5 [35-43,48 ,50 ,51 In summary, 10 articles evaluated the effectiveness of an existing app that was available for use at the time of the evaluation study [38,44,46,47,49,54,56,59,60,68].”
BackgroundO’Dea, B., Han, J., Batterham, P. J., Achilles, M. R., Calear, A. L., Werner‐Seidler, A., Parker, B., Shand, F., & Christensen, H. (2020) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13294 · Cited by 77 · cited 7×
“It was also noted that ongoing updates are likely needed to remain relevant and competitive with other market products, which involves undertaking regular market scans and content refinement to ensure the product remains well positioned and effective in an increasingly saturated market [61].”
BackgroundInfluentialBendtsen, M., Müssener, U., Linderoth, C., & Thomas, K. (2020) · JMIR mhealth and uhealth · doi:10.2196/17208 · Cited by 71 · cited 1×
“Of the 20 apps sustained after development, several are available noncommercially and freely to users in local contexts, including in mental health services [37,42], for university students [36,59], and for corporate organizations [63].”
BackgroundLevin, M. E., Hicks, E. T., & Krafft, J. (2022) · Journal of American College Health · doi:10.1080/07448481.2020.1728281 · Cited by 65
Fish, M. T., & Saul, A. D. (2019) · Simulation & Gaming · doi:10.1177/1046878119851821 · Cited by 61 · cited 5×
“%) evaluated an app still available on the market: Headspace [44,46,47,68], Calm [49], Pacifica/Sanvello [38], Smiling Mind [47], Stop Breathe Think [56], and Thrive [59].”
BackgroundMethodInfluentialLevin, M. E., Haeger, J., & Cruz, R. A. (2019) · Mindfulness · doi:10.1007/s12671-018-1004-2 · Cited by 59 · cited 1×
“The sustainability and penetration of apps is contingent on projects being funded or grants being successful, and even when funding is secured, it is often time limited [42,67,75].”
BackgroundWozney, L., McGrath, P. J., Gehring, N. D., Bennett, K., Huguet, A., Hartling, L., Dyson, M. P., Soleimani, A., & Newton, A. S. (2018) · JMIR Mental Health · doi:10.2196/mental.9655 · Cited by 50
Hill, C., Creswell, C., Vigerland, S., Nauta, M. H., March, S., Donovan, C., Wolters, L., Spence, S. H., Martin, J. L., Wozney, L., et al. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.02.002 · Cited by 44
A randomized controlled trial of a smartphone-based application for the treatment of anxiety
Newman, M. G., Jacobson, N. C., Rackoff, G. N., Bell, M. J., & Taylor, C. B. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1790688 · Cited by 37
Levin, M. E., Haeger, J., An, W., & Twohig, M. P. (2018) · Cognitive Therapy and Research · doi:10.1007/s10608-018-9944-3 · Cited by 36 · cited 1×
“…[55] have focused their efforts on developing app prototypes rapidly using easy-to-use website platforms to customize intervention content and develop generalizable knowledge about principles and processes that work in mobile apps rather than building and developing new apps from scratch [55,57].”
BackgroundThe efficiency model of support and the creation of digital apothecaries.
Muñoz, R. F. (2017) · Clinical Psychology: Science and Practice · doi:10.1111/cpsp.12174 · Cited by 25 · cited 1×
“Although many of these apps can be accessed directly by individuals in the commercial app marketplace as self-care tools, they are also playing an increasing role in clinical services, supplementing or enhancing traditional interventions [3].”
BackgroundKajitani, K., Higashijima, I., Kaneko, K., Matsushita, T., Fukumori, H., & Kim, D. (2020) · PLOS ONE · doi:10.1371/journal.pone.0239592 · Cited by 22
Cited by
The Lancet Psychiatry Commission on youth mental health
McGorry, P. D., Mei, C., Dalal, N., Alvarez-Jimenez, M., Blakemore, S.-J., Browne, V., Dooley, B., Hickie, I. B., Jones, P. B., McDaid, D., et al. (2024) · The Lancet Psychiatry · doi:10.1016/s2215-0366(24)00163-9 · Cited by 663
Torous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025) · World Psychiatry · doi:10.1002/wps.21299 · Cited by 323
Roach, A., Cullinan, S., Shafran, R., Heyman, I., & Bennett, S. (2023) · British Medical Bulletin · doi:10.1093/bmb/ldad001 · Cited by 21
“Organizational factors are often fixed and difficult to change, however, identifying barriers such as cost can help inform researchers at intervention development stage, ensuring that implementation is considered from the start.(40) Engaging wider teams and networks rather than reliance on individual members of staff…”
Background
Metadata
- Authors
- Bear, H. A., Ayala Nunes, L., DeJesus, J., Liverpool, S., Moltrecht, B., Neelakantan, L., Harriss, E., Watkins, E., & Fazel, M.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2022-11-09
- Volume
- 24
- Issue
- 11
- Pages
- e40347
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W4308726987
- Cited by
- 51
- References
- 49
- FWCI
- 2.34
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.