Scaling evidence-based treatments through digital mental health.
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
A large number of Internet websites, mobile apps, wearable devices, and video games that are hoped to improve behavioral, cognitive, and affective targets associated with mental health have been developed. Although technology provides the promising potential to help spread evidence-based mental health treatments, that potential is still largely unrealized. The vast majority of technology-based mental health products, deemed digital mental health interventions (DMHIs), have not been tested and show little fidelity to evidence-based treatments. Furthermore, best practices around the use of technologies in clinical services are underdeveloped and few clinicians have been trained to integrate these tools in their practice. We present an overview of DMHIs and discuss key aspects related to their implementation. We organize these aspects according to the Consolidating Framework for Implementation Research, a framework that identifies constructs related to effective implementation. These constructs are categorized into 5 domains including characteristics of DMHIs, the individuals involved, associated processes, the inner setting, and the outer setting. Our goal is to highlight key areas of consideration for leveraging technology to support the implementation of evidence-based treatments and to emphasize challenges and opportunities that come from using technology to scale evidence-based mental health treatments. (PsycInfo Database Record (c) 2020 APA, all rights reserved).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“We organize these aspects according to the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009).”
MethodPowell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015) · Implementation Science · doi:10.1186/s13012-015-0209-1 · Cited by 4,963
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 1×
“Newer self-guided DMHIs have leveraged more advanced technologies, for example creating chatbots that mirror therapeutic interactions (Fitzpatrick, Darcy, & Vierhile, 2017).”
BackgroundLinardon, J., Cuijpers, P., Carlbring, P., Messer, M., & Fuller‐Tyszkiewicz, M. (2019) · World Psychiatry · doi:10.1002/wps.20673 · Cited by 860
Proudfoot, J., Ryden, C., Everitt, B., Shapiro, D. A., Goldberg, D., Mann, A., Tylee, A., Marks, I., & Gray, J. A. (2004) · British Journal of Psychiatry · doi:10.1192/bjp.185.1.46 · Cited by 516 · cited 1×
“Early self-guided DMHIs attempted to investigate whether EBTs could be translated into digital forms while maintaining efficacy (e.g., Proudfoot et al., 2004; Christensen et al., 2005); these interventions often mirrored self-help books but updated with the latest technologies.”
BackgroundMobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438
Gilbody, S., Littlewood, E., Hewitt, C., Brierley, G., Tharmanathan, P., Araya, R., Barkham, M., Bower, P., Cooper, C., Gask, L., et al. (2015) · BMJ · doi:10.1136/bmj.h5627 · Cited by 409
The Use and Effectiveness of Mobile Apps for Depression: Results From a Fully Remote Clinical Trial
Arean, P. A., Hallgren, K. A., Jordan, J. T., Gazzaley, A., Atkins, D. C., Heagerty, P. J., & Anguera, J. A. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.6482 · Cited by 376
Mohr, D. C., Weingardt, K. R., Reddy, M., & Schueller, S. M. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600541 · Cited by 365 · cited 1×
“Barriers to successful clinical integration often result from systems and organizations thinking of DMHIs as products, rather than services (Mohr et al., 2017), and as previously discussed, lack of knowledge about how to use these tools in one’s practices.”
BackgroundSelf-citationTorous, J., & Roberts, L. W. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0262 · Cited by 362
Self-citationUsing science to sell apps: Evaluation of mental health app store quality claims
Larsen, M. E., Huckvale, K., Nicholas, J., Torous, J., Birrell, L., Li, E., & Reda, B. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0093-1 · Cited by 334
Self-citationA Systematic Review of Cognitive Behavioral Therapy and Behavioral Activation Apps for Depression
Huguet, A., Rao, S., McGrath, P. J., Wozney, L., Wheaton, M., Conrod, J., & Rozario, S. (2016) · PLOS ONE · doi:10.1371/journal.pone.0154248 · Cited by 328 · cited 1×
“A review of over 100 apps proposing to offer cognitive-behavioral therapy (CBT) or behavioral activation (BA) found that only 10% contained features consistent with CBT and BA (Huguet et al., 2016).”
BackgroundWright, J. H., Wright, A. S., Albano, A. M., Basco, M. R., Goldsmith, L. J., Raffield, T., & Otto, M. W. (2005) · American Journal of Psychiatry · doi:10.1176/appi.ajp.162.6.1158 · Cited by 319
Internet interventions: Past, present and future
Andersson, G. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.03.008 · Cited by 315 · cited 1×
“The efficacy of DMHIs has been shown for a multitude of mental health conditions (Andersson, 2018), spanning common mental health issues such as depression and anxiety (Karyotaki et al.”
BackgroundHuckvale, K., Torous, J., & Larsen, M. E. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.2542 · Cited by 312 · cited 1×
“…how many DMH interventions are monetizing data coming from these products, a recent analysis of depression and smoking cessation apps found that 92% were transmitting data to thirty-parties, and some did not contain privacy policies to disclose this information (Huckvale, Torous, & Larsen, 2019).”
BackgroundSelf-citationKaryotaki, E., Ebert, D. D., Donkin, L., Riper, H., Twisk, J., Burger, S., Rozental, A., Lange, A., Williams, A. D., Zarski, A. C., et al. (2018) · Clinical Psychology Review · doi:10.1016/j.cpr.2018.06.007 · Cited by 303 · cited 1×
“For example, research broadly supports that DMHIs provided with human support are effective with large and robust rates of treatment response and remission (Karyotaki et al., 2018).”
BackgroundTorous, J., Friedman, R., & Keshavan, M. (2014) · JMIR mHealth and uHealth · doi:10.2196/mhealth.2994 · Cited by 287 · cited 1×
“Smartphone ownership in the United States is at 81% (Pew, 2019), with many studies of mental health consumers demonstrating similar rates (Lipschitz et al., 2019; Torous, Friedman, & Keshavan, 2014).”
BackgroundSelf-citationSmartphone Apps for Schizophrenia: A Systematic Review
Firth, J., & Torous, J. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.4930 · Cited by 268
Self-citationUsability of Commercially Available Mobile Applications for Diverse Patients
Sarkar, U., Gourley, G. I., Lyles, C. R., Tieu, L., Clarity, C., Newmark, L., Singh, K., & Bates, D. W. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3771-6 · Cited by 257
Using evidence-based internet interventions to reduce health disparities worldwide.
Muñoz, R. F. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1463 · Cited by 242 · cited 1×
“Self-guided DMHIs have the potential to be delivered at scale, with marginally increasing costs for each additional individual who accesses and benefits from such an intervention (Muñoz, 2010).”
BackgroundVan Ameringen, M., Turna, J., Khalesi, Z., Pullia, K., & Patterson, B. (2017) · Depression and Anxiety · doi:10.1002/da.22657 · Cited by 234 · cited 1×
“The vast majority of DMHIs available to consumers are “self-guided” in that they include no human support and are intended mostly for self-help purposes (van Ameringen et al., 2017).”
BackgroundAnxiety: There is an app for that. A systematic review of anxiety apps: SUCALA et al.
Sucala, M., Cuijpers, P., Muench, F., Cardoș, R., Soflau, R., Dobrean, A., Achimas-Cadariu, P., & David, D. (2017) · Depression and Anxiety · doi:10.1002/da.22654 · Cited by 232 · cited 1×
“However, most of the products that have been evaluated in efficacy trials are not publicly available, and the majority of the publicly available products have not been directly evaluated in research studies (Larsen, Nicholas, & Christensen, 2016; Sucala et al., 2017).”
BackgroundChristensen, H., Griffiths, K., Groves, C., & Korten, A. (2006) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/j.1440-1614.2006.01743.x · Cited by 201
Torous, J., Wisniewski, H., Liu, G., & Keshavan, M. (2018) · JMIR Mental Health · doi:10.2196/11715 · Cited by 201 · cited 3×
“The majority of consumers who own a smartphone may not actually download or use health app (Torous et al, 2018).”
BackgroundSelf-citationVis, C., Mol, M., Kleiboer, A., Bührmann, L., Finch, T., Smit, J., & Riper, H. (2018) · JMIR Mental Health · doi:10.2196/mental.9769 · Cited by 183 · cited 1×
“Reliable access to technologies has been found to be a specific determinant of implementation of DMHIs (Vis et al., 2018).”
BackgroundReviewing the data security and privacy policies of mobile apps for depression
O'Loughlin, K., Neary, M., Adkins, E. C., & Schueller, S. M. (2019) · Internet Interventions · doi:10.1016/j.invent.2018.12.001 · Cited by 179
Self-citationSchueller, S. M., Neary, M., O'Loughlin, K., & Adkins, E. C. (2018) · Journal of Medical Internet Research · doi:10.2196/10141 · Cited by 167
Self-citationHermes, E. D., Lyon, A. R., Schueller, S. M., & Glass, J. E. (2019) · Journal of Medical Internet Research · doi:10.2196/11752 · Cited by 149
Self-citationBaumel, A., Edan, S., & Kane, J. M. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/ibz147 · Cited by 143 · cited 2×
“As many people abandon apps after less than one session (Baumel et al., 2019), an initial review often provides enough information to understand the features that might promote sustained use.”
BackgroundBucci, S., Berry, N., Morris, R., Berry, K., Haddock, G., Lewis, S., & Edge, D. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00344 · Cited by 124 · cited 1×
“Clinicians, however, do report some concerns of such resources, especially whether they properly handle issues of data security and clinical safety (Bucci et al., 2019).”
BackgroundMohr, D. C., Schueller, S. M., Tomasino, K. N., Kaiser, S. M., Alam, N., Karr, C., Vergara, J. L., Gray, E. L., Kwasny, M. J., & Lattie, E. G. (2019) · Journal of Medical Internet Research · doi:10.2196/13609 · Cited by 116 · cited 2×
“…found efficacious are made publicly available, the public versions may lack certain aspects that were present in the efficacy trials including human support (e.g., IntelliCare, Mohr et al., 2019), regular assessments (e.g., Schlosser et al., 2016), or monetary incentives (Arean et al., 2016).”
BackgroundSelf-citationBerry, N., Bucci, S., & Lobban, F. (2017) · JMIR Mental Health · doi:10.2196/mental.8311 · Cited by 112
Schlosser, D., Campellone, T., Kim, D., Truong, B., Vergani, S., Ward, C., & Vinogradov, S. (2016) · JMIR Research Protocols · doi:10.2196/resprot.5450 · Cited by 103 · cited 1×
“…found efficacious are made publicly available, the public versions may lack certain aspects that were present in the efficacy trials including human support (e.g., IntelliCare, Mohr et al., 2019), regular assessments (e.g., Schlosser et al., 2016), or monetary incentives (Arean et al., 2016).”
BackgroundExploring mental health providers' interest in using web and mobile-based tools in their practices
Schueller, S. M., Washburn, J. J., & Price, M. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.004 · Cited by 98 · cited 1×
“In general, studies have shown that many clinicians hold favorable views of DMHIs (Schueller et al., 2016; Miller, Kuhn, Yu, et al., 2019; Lattie et al., 2020).”
ResultSelf-citationThe Complexity of Mental Health App Privacy Policies: A Potential Barrier to Privacy
Powell, A., Singh, P., & Torous, J. (2018) · JMIR mHealth and uHealth · doi:10.2196/mhealth.9871 · Cited by 65
Self-citationReimbursement of Apps for Mental Health: Findings From Interviews
Powell, A. C., Bowman, M. B., & Harbin, H. T. (2019) · JMIR Mental Health · doi:10.2196/14724 · Cited by 35
Cited by
Torous, J., Bucci, S., Bell, I. H., Kessing, L. V., Faurholt‐Jepsen, M., Whelan, P., Carvalho, A. F., Keshavan, M., Linardon, J., & Firth, J. (2021) · World Psychiatry · doi:10.1002/wps.20883 · Cited by 1,199
Self-citationTorous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025) · World Psychiatry · doi:10.1002/wps.21299 · Cited by 323
Self-citation“It happened to be the perfect thing”: experiences of generative AI chatbots for mental health
Siddals, S., Torous, J., & Coxon, A. (2024) · npj Mental Health Research · doi:10.1038/s44184-024-00097-4 · Cited by 241
Self-citationEngagement in digital interventions.
Nahum-Shani, I., Shaw, S. D., Carpenter, S. M., Murphy, S. A., & Yoon, C. (2022) · American Psychologist · doi:10.1037/amp0000983 · Cited by 177
Providing Human Support for the Use of Digital Mental Health Interventions: Systematic Meta-review
Werntz, A., Amado, S., Jasman, M., Ervin, A., & Rhodes, J. E. (2023) · Journal of Medical Internet Research · doi:10.2196/42864 · Cited by 158
Metadata
- Authors
- Schueller, S. M., & Torous, J.
- Venue
- American Psychologist
- Published
- 2020-11
- Volume
- 75
- Issue
- 8
- Pages
- 1093-1104
- ISSN
- 0003-066X (ISSN-L)
- 1935-990X
- OpenAlex
- W3110201756
- Cited by
- 175
- References
- 36
- FWCI
- 14.05
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.