Using science to sell apps: Evaluation of mental health app store quality claims
Open accessAbstract
Despite the emergence of curated app libraries for mental health apps, personal searches by consumers remain a common method for discovering apps. App store descriptions therefore represent a key channel to inform consumer choice. This study examined the claims invoked through these app store descriptions, the extent to which scientific language is used to support such claims, and the corresponding evidence in the literature. Google Play and iTunes were searched for apps related to depression, self-harm, substance use, anxiety, and schizophrenia. The descriptions of the top-ranking, consumer-focused apps were coded to identify claims of acceptability and effectiveness, and forms of supporting statement. For apps which invoked ostensibly scientific principles, a literature search was conducted to assess their credibility. Seventy-three apps were coded, and the majority (64%) claimed effectiveness at diagnosing a mental health condition, or improving symptoms, mood or self-management. Scientific language was most frequently used to support these effectiveness claims (44%), although this included techniques not validated by literature searches (8/24 = 33%). Two apps described low-quality, primary evidence to support the use of the app. Only one app included a citation to published literature. A minority of apps (14%) described design or development involving lived experience, and none referenced certification or accreditation processes such as app libraries. Scientific language was the most frequently invoked form of support for use of mental health apps; however, high-quality evidence is not commonly described. Improved knowledge translation strategies may improve the adoption of other strategies, such as certification or lived experience co-design.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other reviews concluded that the quality of evidence supporting the efficacy of mental health apps is insufficient.
“Other reviews have evaluated mental health apps more broadly, concluding that the quality of evidence supporting their efficacy is insufficient (Larsen et al. 2019; Wisniewski et al. 2019)”
From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as Larsen et al. 2019
The Discussion (page unavailable) says that only two of the 73 apps examined provided direct evidence associated with app use, and that even this fell short of high-quality evidence. That supports concern about the evidence for the apps reviewed, but not a general conclusion that evidence for mental health app efficacy is insufficient; the Introduction notes reviews finding apps effective for depression and anxiety. (LLM, conf=0.86)
Experimental — accuracy not yet measured55% of the apps most commonly used by individuals with BD have published evidence.
“Whilst reviews of the broader BD app landscape report very low incidence of scientific evidence for available apps (Larsen et al. 2019), 55% of the apps most commonly used by individuals with BD have published evidence (albeit, with the limitations discussed above).”
From Evaluating the quality, safety, and functionality of commonly used smartphone apps for bipolar disorder mood and sleep self-management (2022) · cites this paper as Larsen et al. 2019
The citation is attached to the first clause, about low scientific-evidence availability, not to the citing authors’ 55% figure. The Discussion reports that only two of 73 apps provided direct evidence associated with app use, supporting the low-incidence point. However, the Methods’ search strategy covers five mental health conditions and does not specifically examine the broader bipolar-disorder app landscape. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The empirical status of cognitive-behavioral therapy: A review of meta-analyses
Butler, A. C., Chapman, J. E., Forman, E. M., & Beck, A. T. (2006) · Clinical Psychology Review · doi:10.1016/j.cpr.2005.07.003 · Cited by 3,215 · see citing works
Estimating the true global burden of mental illness
Vigo, D., Thornicroft, G., & Atun, R. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(15)00505-2 · Cited by 2,494 · see citing works
Plummer, F., Manea, L., Trepel, D., & McMillan, D. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2015.11.005 · Cited by 1,882 · see citing works
Gilbody, S., Richards, D., Brealey, S., & Hewitt, C. (2007) · Journal of General Internal Medicine · doi:10.1007/s11606-007-0333-y · Cited by 1,290 · see citing works
Firth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097 · see citing works
Self-citationTorous, J., Nicholas, J., Larsen, M. E., Firth, J., & Christensen, H. (2018) · Evidence Based Mental Health · doi:10.1136/eb-2018-102891 · Cited by 790 · see citing works
Self-citationFirth, J., Torous, J., Nicholas, J., Carney, R., Rosenbaum, S., & Sarris, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.04.046 · Cited by 675 · see citing works
Self-citationMobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438 · see citing works
Self-citationMohr, D. C., Lyon, A. R., Lattie, E. G., Reddy, M., & Schueller, S. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7725 · Cited by 432 · see citing works
A Systematic Review of Cognitive Behavioral Therapy and Behavioral Activation Apps for Depression
Huguet, A., Rao, S., McGrath, P. J., Wozney, L., Wheaton, M., Conrod, J., & Rozario, S. (2016) · PLOS ONE · doi:10.1371/journal.pone.0154248 · Cited by 328 · see citing works
Psychosocial interventions for self-harm in adults
Hawton, K. E., Witt, K., Salisbury, T. T., Arensman, E., Gunnell, D., Hazell, P., Townsend, E., & van Heeringen, K. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012189 · Cited by 314 · see citing works
A systematic assessment of smartphone tools for suicide prevention
Larsen, M. E., Nicholas, J., & Christensen, H. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152285 · Cited by 289 · see citing works
Apps for asthma self-management: a systematic assessment of content and tools
Huckvale, K., Car, M., Morrison, C. P., & Car, J. (2012) · BMC Medicine · doi:10.1186/1741-7015-10-144 · Cited by 252 · see citing works
Self-citationTighe, J., Shand, F., Ridani, R., Mackinnon, A., De La Mata, N., & Christensen, H. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-013518 · Cited by 227 · see citing works
Many Mobile Health Apps Target High-Need, High-Cost Populations, But Gaps Remain
Singh, K., Drouin, K., Newmark, L. P., Lee, J., Faxvaag, A., Rozenblum, R., Pabo, E. A., Landman, A., Klinger, E., & Bates, D. W. (2016) · Health Affairs · doi:10.1377/hlthaff.2016.0578 · Cited by 201 · see citing works
Harm reduction -a systematic review on effects of alcohol reduction on physical and mental symptoms
Charlet, K., & Heinz, A. (2016) · Addiction Biology · doi:10.1111/adb.12414 · Cited by 184 · see citing works
Torous, J. B., Chan, S. R., Gipson, S. Y.-M. T., Kim, J. W., Nguyen, T.-Q., Luo, J., & Wang, P. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700423 · Cited by 184 · see citing works
Self-citationQuantifying App Store Dynamics: Longitudinal Tracking of Mental Health Apps
Larsen, M. E., Nicholas, J., & Christensen, H. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.6020 · Cited by 177 · see citing works
Self-citationSchueller, S. M., Neary, M., O'Loughlin, K., & Adkins, E. C. (2018) · Journal of Medical Internet Research · doi:10.2196/10141 · Cited by 167 · see citing works
The evolution of mobile apps for asthma: an updated systematic assessment of content and tools
Huckvale, K., Morrison, C., Ouyang, J., Ghaghda, A., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0303-x · Cited by 166 · see citing works
Self-citationNicholas, J., Fogarty, A. S., Boydell, K., & Christensen, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7273 · Cited by 158 · see citing works
Self-citationActive listening: more than just paying attention.
Robertson, K. R. (2005) · PubMed · Cited by 139 · see citing works
Free smoking cessation mobile apps available in Australia: a quality review and content analysis
Thornton, L. K., Quinn, C. A., Birrell, L., Guillaumier, A., Shaw, B., Forbes, E., Deady, M., & KAY‐LAMBKIN, F. J. (2017) · Australian and New Zealand Journal of Public Health · doi:10.1111/1753-6405.12688 · Cited by 93 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [26]OCEBM Levels of Evidence Working Group. The Oxford Levels of Evidence 2 https://www.cebm.net/index.aspx?o=5653 (2016).
Cited by
Sorted by citations
Torous, J., Bucci, S., Bell, I. H., Kessing, L. V., Faurholt‐Jepsen, M., Whelan, P., Carvalho, A. F., Keshavan, M., Linardon, J., & Firth, J. (2021) · World Psychiatry · doi:10.1002/wps.20883 · Cited by 1,199 · see citing works
Self-citationTorous, J., Jän Myrick, K., Rauseo-Ricupero, N., & Firth, J. (2020) · JMIR Mental Health · doi:10.2196/18848 · Cited by 946 · see citing works
“Picking from those different apps is challenging as many offer little protection of user data, make exaggerated claims, may be ineffective, and often are quickly abandoned because of usability issues [18,23,24].”
BackgroundSelf-citationDigital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390 · see citing works
Stein, D. J., Shoptaw, S. J., Vigo, D. V., Lund, C., Cuijpers, P., Bantjes, J., Sartorius, N., & Maj, M. (2022) · World Psychiatry · doi:10.1002/wps.20998 · Cited by 331 · see citing works
Beyond validation: getting health apps into clinical practice
Gordon, W. J., Landman, A., Zhang, H., & Bates, D. W. (2020) · npj Digital Medicine · doi:10.1038/s41746-019-0212-z · Cited by 322 · see citing works
Virtual reality as a clinical tool in mental health research and practice
Bell, I. H., Nicholas, J., Alvarez-Jimenez, M., Thompson, A., & Valmaggia, L. (2020) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2020.22.2/lvalmaggia · Cited by 302 · see citing works
Self-citationResponsible AI—two frameworks for ethical design practice
Peters, D., Vold, K., Robinson, D., & Calvo, R. A. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.2974991 · Cited by 293 · see citing works
“A number of authors within human–computer interaction have articulated some of the ethically loaded socio-technical challenges facing digital mental health developers [28]–[31], [43], [44].”
BackgroundThe Need for a Mental Health Technology Revolution in the COVID-19 Pandemic
Figueroa, C. A., & Aguilera, A. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00523 · Cited by 227 · see citing works
Linardon, J., Torous, J. B., Firth, J., Cuijpers, P. J., Messer, M., & Fuller‐Tyszkiewicz, M. (2024) · World Psychiatry · doi:10.1002/wps.21183 · Cited by 231 · see citing works
“…with depression and anxiety apps, such as privacy violations; possible easy access to ineffective, inaccurate or potentially harmful content 15,16 ; low rates of engagement 17,18 , and exclusion of the potential therapeutic ingredient represented by the personal relationship between a…”
BackgroundSelf-citationCarlo, F. D., Sociali, A., Picutti, E., Pettorruso, M., Vellante, F., Verrastro, V., Martinotti, G., & Giannantonio, M. D. (2020) · International Journal of Clinical Practice · doi:10.1111/ijcp.13716 · Cited by 216 · see citing works
“Through a search on dedicated platforms conducted in 2019, 1435 mobile apps were identified, 449 of which addressing anxiety problems, 450 addressing Depression, 282 addressing Schizophrenia, 124 addressing self-harm and 140 addressing Substance Use Disorders [37].”
Rauschenberg, C., Schick, A., Hirjak, D., Seidler, A., Paetzold, I., Apfelbacher, C., Riedel-Heller, S. G., & Reininghaus, U. (2021) · Journal of Medical Internet Research · doi:10.2196/23365 · Cited by 205 · see citing works
“There are a number of reviews that have concluded that mHealth apps have problematic data-sharing and privacy practices [8,27,28] and that there may not only be a lack of quality of offered content but even harmful intervention components.”
BackgroundInfluentialFiol-deRoque, M. A., Serrano-Ripoll, M. J., Jiménez, R., Zamanillo‐Campos, R., Yáñez, A. M., Bennasar‐Veny, M., Leiva, A., Gervilla, E., Buades, E. G., Garcı́a-Toro, M., et al. (2021) · JMIR mhealth and uhealth · doi:10.2196/27039 · Cited by 203 · see citing works
Bauer, M., Glenn, T., Geddes, J., Gitlin, M., Grof, P., Kessing, L. V., Monteith, S., Faurholt-Jepsen, M., Severus, E., & Whybrow, P. C. (2020) · International Journal of Bipolar Disorders · doi:10.1186/s40345-019-0164-x · Cited by 163 · see citing works
Scaling evidence-based treatments through digital mental health.
Schueller, S. M., & Torous, J. B. (2020) · American Psychologist · doi:10.1037/amp0000654 · Cited by 174 · see citing works
Self-citationRauschenberg, C., Schick, A., Goetzl, C., Roehr, S., Riedel-Heller, S. G., Koppe, G., Durstewitz, D., Krumm, S., & Reininghaus, U. (2021) · European Psychiatry · doi:10.1192/j.eurpsy.2021.17 · Cited by 158 · see citing works
Digital interventions in mental health: An overview and future perspectives
Löchner, J., Carlbring, P., Schuller, B. W., Torous, J. B., & Sander, L. (2025) · Internet Interventions · doi:10.1016/j.invent.2025.100824 · Cited by 170 · see citing works
“It becomes problematic when many private companies are less than transparent and methodologically thorough in their processing of personal data, e.g., for commercial and unintended purposes (Larsen et al., 2019).”
Self-citationActionable health app evaluation: translating expert frameworks into objective metrics
Lagan, S., Aquino, P. R., Emerson, M. R., Fortuna, K. L., Walker, R., & Torous, J. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00312-4 · Cited by 146 · see citing works
Self-citationCarter, H., Araya, R., Anjur, K. I., Deng, D., & Naslund, J. A. (2020) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2020.12.016 · Cited by 155 · see citing works
“For instance, over 10,000 mental health apps are available for download via commercial app stores (Torous et al., 2019); though, most of these commercially available apps have no reliable or high-quality scientific evidence as support (Larsen et al., 2019; Lau et al., 2020).”
BackgroundAdvances in the use of virtual reality to treat mental health conditions
Bell, I. H., Pot-Kolder, R., Rizzo, A., Rus-Calafell, M., Cardi, V., Cella, M., Ward, T., Riches, S., Reinoso, M., Thompson, A., et al. (2024) · Nature Reviews Psychology · doi:10.1038/s44159-024-00334-9 · Cited by 140 · see citing works
“With the proliferation of digital health products available directly to consumers, many of which make strong claims without sufficient scientific evidence 148 , there is a need to address growing concerns about ethics and regulation in the digital mental health field.”
Smartphone apps for the treatment of mental health conditions: status and considerations
Huckvale, K., Nicholas, J., Torous, J. B., & Larsen, M. E. (2020) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2020.04.008 · Cited by 137 · see citing works
Self-citationHealth app policy: international comparison of nine countries’ approaches
Essén, A., Stern, A. D., Haase, C. B., Car, J., Greaves, F., Paparova, D., Vandeput, S., Wehrens, R., & Bates, D. W. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00573-1 · Cited by 131 · see citing works
Van Daele, T., Karekla, M., Kassianos, A. P., Compare, A., Haddouk, L., Salgado, J., Ebert, D. D., Trebbi, G., Bernaerts, S., Van Assche, E., et al. (2020) · Journal of Psychotherapy Integration · doi:10.1037/int0000218 · Cited by 122 · see citing works
“Despite the increasing evidence-base for e-mental health, commercially available applications often lack scientific background and are hardly ever validated in research (Anthes, 2016; Larsen et al., 2019).”
BackgroundCognitive-Behavioral Therapy in the Digital Age: Presidential Address
Wilhelm, S., Weingarden, H. M., Ladis, I., Braddick, V., Shin, J. Y., & Jacobson, N. C. (2019) · Behavior Therapy · doi:10.1016/j.beth.2019.08.001 · Cited by 122 · see citing works
“However, when these descriptions referenced specific treatment approaches, only about half described methods supported in the literature—furthermore, only two descriptions cited research about the app itself to back up effectiveness claims (Larsen et al., 2019).”
BackgroundInfluentialZidaru, T., Morrow, E. M., & Stockley, R. (2021) · Health Expectations · doi:10.1111/hex.13299 · Cited by 138 · see citing works
Better, Virtually: the Past, Present, and Future of Virtual Reality Cognitive Behavior Therapy
Lindner, P. (2021) · International Journal of Cognitive Therapy · doi:10.1007/s41811-020-00090-7 · Cited by 120 · see citing works
“Research on consumer smartphone applications for mental health (Larsen et al. 2019; Shen et al. 2015) suggests that few future consumer VR applications that will be released on ordinary digital marketplaces can be expected to be evidence-based and effective.”
BackgroundHuman-Centered Design Approaches in Digital Mental Health Interventions: Exploratory Mapping Review
Vial, S., Boudhraâ, S., & Dumont, M. (2022) · JMIR Mental Health · doi:10.2196/35591 · Cited by 114 · see citing works
“Several issues raised in recent years have not been resolved: there is some distrust in the field that is not served by the lack of empirical validation of its benefit [6-9]; it raises privacy and data security concerns [9,10]; it presents commercial issues (eg, financial interest, user access, advertising) [11,12]; a…”
BackgroundCollege student engagement with mental health apps: analysis of barriers to sustained use
Melcher, J., Camacho, E., Lagan, S., & Torous, J. (2022) · Journal of American College Health · doi:10.1080/07448481.2020.1825225 · Cited by 106 · see citing works
“They are mindful that today anyone can put an app on the app store and make it look legitimate.(35) Survey results demonstrate that students are cognizant of false information with 60% of respondents reporting that accuracy of recommendations from mental health apps was among their top concerns for using these apps.”
BackgroundSelf-citationFriis-Healy, E. A., Nagy, G. A., & Kollins, S. H. (2021) · JMIR Mental Health · doi:10.2196/25456 · Cited by 109 · see citing works
“Reliance on the “wisdom of the crowd” to guide decision making is not advised since, among other reasons, user ratings do not correlate well with clinical utility or quality [27].”
BackgroundAI-Based and Digital Mental Health Apps: Balancing Need and Risk
Hamdoun, S., Monteleone, R., Bookman, T., & Michael, K. (2023) · IEEE Technology and Society Magazine · doi:10.1109/mts.2023.3241309 · Cited by 104 · see citing works
“…optimistic about the use of digital mental health apps, with commercial entities, in particular, touting a range of benefits for self-help services [32], such as the convenience of on-demand access to resources [33], the ability to address a range of mental health concerns holistically using a…”
BackgroundUnderstanding People’s Use of and Perspectives on Mood-Tracking Apps: Interview Study
Schueller, S. M., Neary, M., Lai, J., & Epstein, D. A. (2021) · JMIR Mental Health · doi:10.2196/29368 · Cited by 102 · see citing works
Serrano-Ripoll, M. J., Zamanillo-Campos, R., Fiol-DeRoque, M. A., Castro, A., & Ricci-Cabello, I. (2022) · JMIR mHealth and uHealth · doi:10.2196/29621 · Cited by 96 · see citing works
“The digital market is full of apps designed to improve the mental health of people with depression; however, most of them remain untested in clinical trials and suffer from numerous limitations, such as being designed without content based on the recommendations of experts [11,12].”
BackgroundSeiferth, C., Vogel, L., Aas, B., Brandhorst, I., Carlbring, P., Conzelmann, A., Esfandiari, N., Finkbeiner, M., Hollmann, K., Lautenbacher, H., et al. (2023) · Nature Mental Health · doi:10.1038/s44220-023-00085-1 · Cited by 99 · see citing works
Self-citationRoberts, A., Davenport, T. A., Wong, T., Moon, H.-W., Hickie, I. B., & LaMonica, H. M. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100379 · Cited by 99 · see citing works
“Consistent with previous research (Alyami et al., 2017; Larsen et al., 2019; Sucala et al., 2017; Van Ameringen et al., 2017), the majority of apps and e-tools rated in this study did not meet this standard (mean = 3.43, standard deviation = 0.87).”
ResultInfluentialBennett, C. B., Ruggero, C. J., Sever, A. C., & Yanouri, L. (2020) · Journal of Psychotherapy Integration · doi:10.1037/int0000217 · Cited by 87 · see citing works
Götzl, C., Hiller, S., Rauschenberg, C., Schick, A., Fechtelpeter, J., Fischer-Abaigar, U., Koppe, G., Durstewitz, D., Reininghaus, U., & Krumm, S. (2022) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-022-00522-6 · Cited by 84 · see citing works
“Existing apps available in major app stores rarely use evidence–based frameworks [16–19], frequently do not involve young people in the development [20], and therefore often remain insufficiently tailored to the needs of young users [4, 20, 21].”
BackgroundSmartphone-Enhanced Symptom Management In Psychosis: Open, Randomized Controlled Trial
Lewis, S. W., Ainsworth, J. D., Sanders, C., Stockton-Powdrell, C., Machin, M., Whelan, P., Hopkins, R., He, Z., Applegate, E., Drake, R., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/17019 · Cited by 83 · see citing works
Weber, S., Lorenz, C., & Hemmings, N. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02745 · Cited by 74 · see citing works
“We assessed resilience with the 13-item Resilience Scale (RS13; Leppert et al., 2008), a short form of the Resilience Scale (RS-25; Wagnild and Young, 1993).”
MethodConnolly, S. L., Hogan, T. P., Shimada, S. L., & Miller, C. J. (2020) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-020-00165-4 · Cited by 78 · see citing works
Ethics of Digital Mental Health During COVID-19: Crisis and Opportunities
Martinez‐Martin, N., Dasgupta, I., Carter, A., Chandler, J. A., Kellmeyer, P., Kreitmair, K., Weiss, A. P., & Cabrera, L. Y. (2020) · JMIR Mental Health · doi:10.2196/23776 · Cited by 85 · see citing works
“Furthermore, the majority of consumer apps are not evidence-based and some even contain harmful content [40].”
BackgroundRegulatory considerations to keep pace with innovation in digital health products
Torous, J. B., Stern, A. D., & Bourgeois, F. T. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00668-9 · Cited by 83 · see citing works
Self-citationDigital Psychiatry: Risks and Opportunities for Public Health and Wellbeing
Burr, C., Morley, J., Taddeo, M., & Floridi, L. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.2977059 · Cited by 78 · see citing works
“, “motivational therapist” or “wellness coach”) or hide behind disclaimers to avoid scrutiny [72].”
BackgroundMerchant, R., Torous, J., Rodriguez-Villa, E., & Naslund, J. A. (2020) · Current Opinion in Psychiatry · doi:10.1097/yco.0000000000000626 · Cited by 70 · see citing works
“A 2015 review that assessed 82 commercially available smartphone apps to manage bipolar disorder concluded that in general, the content was not that only 14% of a sample of 73 smartphone apps involved individuals with lived experience in the design or development process [19 & ].”
BackgroundSelf-citationKarkosz, S., Szymański, R., Sanna, K., & Michałowski, J. M. (2023) · JMIR Formative Research · doi:10.2196/47960 · Cited by 112 · see citing works
“Nevertheless, applications that are available on the market often lack appropriate scientific evidence of feasibility or efficacy [4].”
BackgroundRodriguez-Villa, E., & Torous, J. B. (2019) · BMC Medicine · doi:10.1186/s12916-019-1447-x · Cited by 71 · see citing works
“This lack of transparency, complicated by unmeasured and unfounded claims of many apps [25], makes evaluation cumbersome and time consuming.”
BackgroundSelf-citationThe Debate on the Ethics of AI in Health Care: a Reconstruction and Critical Review
Morley, J., Machado, C., Burr, C., Cowls, J., Taddeo, M., & Floridi, L. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3486518 · Cited by 66 · see citing works
Psychiatry in the Digital Age: A Blessing or a Curse?
Roth, C. B., Papassotiropoulos, A., Brühl, A. B., Lang, U. E., & Huber, C. G. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18168302 · Cited by 71 · see citing works
Is There an App for That?: Ethical Issues in the Digital Mental Health Response to COVID-19
Skorburg, J. A., & Yam, J. (2021) · AJOB Neuroscience · doi:10.1080/21507740.2021.1918284 · Cited by 62 · see citing works
Determinant Factors of Public Acceptance of Stress Management Apps: Survey Study
Apolinário-Hagen, J., Hennemann, S., Fritsche, L., Drüge, M., & Breil, B. (2019) · JMIR Mental Health · doi:10.2196/15373 · Cited by 63 · see citing works
“However, the evidence base for the quality and efficacy of most mental health apps is limited [112], even among those mental health apps that claim to be effective [113].”
Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis
Day, S., Shah, V., Kaganoff, S., Powelson, S., & Mathews, S. C. (2022) · Journal of Medical Internet Research · doi:10.2196/37677 · Cited by 57 · see citing works
Hong, R. H., Murphy, J., Michalak, E. E., Chakrabarty, T., Wang, Z., Parikh, S. V., Culpepper, L., Yatham, L. N., Lam, R. W. M., & Chen, J. (2021) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s283731 · Cited by 58 · see citing works
Nwe, K., Larsen, M. E., Nelissen, N., & Wong, D. (2020) · Journal of Medical Internet Research · doi:10.2196/17457 · Cited by 56 · see citing works
“To date, the evidence for their effectiveness is varied [4-6] and, in general, published evidence is extremely sparse [7].”
BackgroundSelf-citationExploring medical students' views on digital mental health interventions: A qualitative study
Dederichs, M., Weber, J., Pischke, C. R., Angerer, P., & Apolinário-Hagen, J. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100398 · Cited by 50 · see citing works
Tark, R., Metelitsa, M., Akkermann, K., Saks, K., Mikkel, S., & Haljas, K. (2019) · JMIR Serious Games · doi:10.2196/13776 · Cited by 48 · see citing works
“Similarly, another recent review brought out the mental health app efficacy evaluation as a concern, showing that 38% of app store descriptions included phrases related to claims of effectiveness, whereas only less than 3% provided scientific evidence for such declarations [24].”
ResultShidhaye, R. R. (2023) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2023.101661 · Cited by 53 · see citing works
Digital Health COVID-19 Impact Assessment: Lessons Learned and Compelling Needs
Microsoft, Lee, P., Abernethy, A., Verily, Shaywitz, D., HealthTech, A., Gundlapalli, A., Prevention, U. S. C. F. D. C. A., Weinstein, J., Healthcare, M., et al. (2022) · NAM Perspectives · doi:10.31478/202201c · Cited by 49 · see citing works
“eff ectiveness through scientifi c phrasing, but only two provided direct evidence from a trial [51].”
BackgroundIndividual differences in receptivity to scientific bullshit
Evans, A. M., Sleegers, W. W. A., & Mlakar, Ž. (2020) · Judgment and Decision Making · doi:10.1017/s1930297500007191 · Cited by 46 · see citing works
The Future of Peer Support in Digital Psychiatry: Promise, Progress, and Opportunities
Fortuna, K. L., Venegas, M. D. R. A., Umucu, E., Mois, G., Walker, R., & Brooks, J. (2019) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-019-00179-7 · Cited by 46 · see citing works
Spadaro, B., Martin-Key, N. A., & Bahn, S. (2021) · Journal of Medical Internet Research · doi:10.2196/27507 · Cited by 45 · see citing works
“…commonly downloaded mental health apps mainly fall under the umbrella of well-being apps offering relaxation, meditation, or mindfulness skills rather than validated treatments, and that the scientific language used to support claims often lacks corresponding evidence in the literature [28,29].”
Research Trends on Mobile Mental Health Application for General Population: A Scoping Review
Hwang, W. J., Ha, J. S., & Kim, M. J. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18052459 · Cited by 47 · see citing works
Catuara-Solarz, S., Skorulski, B., Estella-Aguerri, I., Avella-Garcia, C. B., Shepherd, S., Stott, E., Hemmings, N. R., de Villa, A. R., Schulze, L., & Dix, S. (2021) · JMIR mhealth and uhealth · doi:10.2196/30976 · Cited by 44 · see citing works
“Although some apps have been shown to have positive effects on mental well-being [10,12,13], concerns remain regarding the level of credibility and the robustness of evidence underlying the majority of the thousands of available apps on the market [10,14].”
Baumel, A., Torous, J. B., Edan, S., & Kane, J. M. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.05.011 · Cited by 40 · see citing works
Self-citationTools for Wellbeing-Supportive Design: Features, Characteristics, and Prototypes
Peters, D., Ahmadpour, N., & Calvo, R. A. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4030040 · Cited by 41 · see citing works
“Taking digital mental health as a cautionary example, failures of rigor have led to an exploding industry of mental health tools that lack evidence, inclusivity, and effectiveness (and at worst, cause harm) [44].”
BackgroundFeldman, N., Back, D., BOLAND, R. J., & Torous, J. B. (2021) · Archives of Women s Mental Health · doi:10.1007/s00737-021-01138-z · Cited by 40 · see citing works
“In terms of the marketplace and app stores, increasing research suggests that apps available to consumers are often untested (Larsen et al. 2019) and may present privacy risks that may expose sensitive mental health data (Huckvale et al.”
BackgroundSelf-citationFagernäs, S., Hamilton, W., Espinoza, N., Miloff, A., Carlbring, P., & Lindner, P. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100370 · Cited by 38 · see citing works
“…past research on consumer smartphone applications, which have been found to vary considerably in both use of evidence-based therapeutic components (Shen et al., 2015), direct empirical support (Linardon et al., 2019) and use of scientific language to support these claims (Larsen et al., 2019).”
Backgroundd'Halluin, A., Costa, M., Morgiève, M., & Sebbane, D. (2023) · Journal of Medical Internet Research · doi:10.2196/43102 · Cited by 34 · see citing works
“Although many of these apps were evidence informed and were inspired by scientifically validated techniques and methods, very few were evidence based (ie, the subject of rigorous scientific studies demonstrating their effectiveness and reliability [10,11]).”
Gasteiger, N., Dowding, D., Norman, G., McGarrigle, L., Eost‐Telling, C., Jones, D. J., Vercell, A., Ali, S. M., & O’Connor, S. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-073283 · Cited by 37 · see citing works
Satre, D. D., Meacham, M. C., Asarnow, L. D., Fisher, W. S., Fortuna, L. R., & Iturralde, E. (2021) · American Journal of Health Promotion · doi:10.1177/08901171211055314 · Cited by 35 · see citing works
“Providers and health systemsmust contendwith a shifting landscape of available apps.(68) The generally unregulated status of mental health apps(69) and the unclear value of app store user reviews have sparked new approaches for endorsing apps and cataloguing recommendations.”
BackgroundHrynyschyn, R., & Dockweiler, C. (2021) · JMIR mHealth and uHealth · doi:10.2196/24703 · Cited by 35 · see citing works
“Currently, thousands of mental health apps are available in the market, and these apps are designed for a brief frequent use throughout the day [24].”
BackgroundGenerating value with mental health apps
Powell, A. C., Torous, J. B., Firth, J., & Kaufman, K. R. (2020) · BJPsych Open · doi:10.1192/bjo.2019.98 · Cited by 33 · see citing works
Self-citationCOVID-19 has inspired global healthcare innovation
Palanica, A., & Fossat, Y. (2020) · Canadian Journal of Public Health · doi:10.17269/s41997-020-00406-2 · Cited by 33 · see citing works
Digital technology for addressing cognitive impairment in recent-onset psychosis: A perspective
Bell, I. H., Pot-Kolder, R., Wood, S. J., Nelson, B., Acevedo, N., Stainton, A., Nicol, K., Kean, J. D., Bryce, S., Bartholomeusz, C. F., et al. (2022) · Schizophrenia Research Cognition · doi:10.1016/j.scog.2022.100247 · Cited by 33 · see citing works
“doscience and scientific jargon to make unsubstantiated claims of effectiveness (Larsen et al., 2019).”
Self-citationCaiani, E. G., Kemps, H., Hoogendoorn, P., Asteggiano, R., Böhm, A., Borregaard, B., Boriani, G., Rocca, H. B., Casado-Arroyo, R., Castelletti, S., et al. (2024) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae042 · Cited by 31 · see citing works
“17,18 Access to mHealth solutions, including wearables such as smart-watches, is further complicated by the fact that they can assume the role of a lifestyle gadget and/or that of an MD, depending on the model and the country in which they are sold.”
BackgroundLeong, Q. Y., Sridhar, S., Blasiak, A., Tadeo, X., Yeo, G., Remus, A., & Ho, D. (2021) · Journal of Medical Internet Research · doi:10.2196/27388 · Cited by 34 · see citing works
“Instead of objectively informing about the evidence, the function of the app description is to appeal to consumers and drive the number of app downloads, independent of the platform’s objective quality [187].”
BackgroundBowie-DaBreo, D., Sas, C., Iles‐Smith, H., & Sünram‐Lea, S. I. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517498 · Cited by 36 · see citing works
“There is therefore need for more evidence on the use of apps for depression in the wild to better understand user experiences and implications for design to improve the quality and acceptance of these apps.”
Schick, A., Paetzold, I., Rauschenberg, C., Hirjak, D., Banaschewski, T., Meyer-Lindenberg, A., Boehnke, J. R., Boecking, B., & Reininghaus, U. (2021) · JMIR Research Protocols · doi:10.2196/27462 · Cited by 30 · see citing works
“However, most mHealth apps that are currently available in major app stores are not evidence-based, often use problematic data sharing and privacy practices, and sometimes contain harmful content [34-36].”
Critical review of self‐diagnosis of mental health conditions using artificial intelligence
Wimbarti, S., Kairupan, B. H. R., & Tallei, T. E. (2024) · International Journal of Mental Health Nursing · doi:10.1111/inm.13303 · Cited by 35 · see citing works
Selaskowski, B., Reiland, M., Schulze, M., Aslan, B., Kannen, K., Wiebe, A., Wallbaum, T., Boll, S., Lux, S., Philipsen, A., et al. (2023) · BJPsych Open · doi:10.1192/bjo.2023.573 · Cited by 32 · see citing works
Haaf, R., Vock, P., Wächtershäuser, N., Correll, C. U., Köhler, S., & Klein, J. P. (2024) · Der Nervenarzt · doi:10.1007/s00115-023-01587-0 · Cited by 30 · see citing works
Koch, E. D., Moukhtarian, T. R., Skirrow, C., Bozhilova, N. S., Asherson, P. J., & Ebner‐Priemer, U. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.06.002 · Cited by 30 · see citing works
Is the world ready for ChatGPT therapists?
Graber-Stiehl, I. (2023) · Nature · doi:10.1038/d41586-023-01473-4 · Cited by 32 · see citing works
Building Better Schools with Evidence-based Policy
Allen, K., Reupert, A., & Oades, L. G. (2021) · Routledge eBooks · doi:10.4324/9781003025955 · Cited by 31 · see citing works
Naslund, J. A., & Aschbrenner, K. A. (2021) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2021.100227 · Cited by 29 · see citing works
“It is worrisome that so few of these commercially available apps are supported with reliable or high-quality scientific evidence, and in the most egregious examples even report false or misleading claims of clinical effectiveness (Larsen et al., 2019; Lau et al., 2020).”
Bautista, J., & Schueller, S. M. (2022) · JMIR Mental Health · doi:10.2196/43942 · Cited by 28 · see citing works
“Although there are mental health apps that are effective, many mental health apps are not evidence based [38].”
Tong, A. C. Y., Wong, K. T. Y., Chung, W. W. T., & Mak, W. W. S. (2025) · Journal of Medical Internet Research · doi:10.2196/70436 · Cited by 28 · see citing works
“Many of these tools are not empirically tested or are misinformed [59-61].”
Recommendations for Successful Development and Implementation of Digital Health Technology Tools
Loo, R. T. J., Nasta, F., Macchi, M., Baudot, A., Burstein, F., Bove, R. M., Greve, M., Fröhlich, H., Khalid, S., Küderle, A., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/56747 · Cited by 28 · see citing works
“Larsen et al [70] found that among 73 top-ranked mental health apps, only 2 provided primary evidence, 1 cited published literature, and 14% described development with lived experience.”
Self-citationBarnett, S., Huckvale, K., Christensen, H. M., Venkatesh, S., Mouzakis, K., & Vasa, R. (2019) · Journal of Medical Internet Research · doi:10.2196/16399 · Cited by 28 · see citing works
“…efforts to build sensing and intervention apps, many appear to have limited potential for clinical translation because they are not backed by evidence of efficacy [28], or do not clearly satisfy clinical quality [29] and safety expectations (such as those concerning diagnostic accuracy [30]).”
BackgroundSelf-citationBowie-DaBreo, D., Sünram‐Lea, S. I., Sas, C., & Iles‐Smith, H. (2020) · JMIR Formative Research · doi:10.2196/14988 · Cited by 26 · see citing works
“…authors commented that although some apps seemed to be evidence informed, reflecting some theoretical principles and strategies, the apps did not demonstrate high fidelity to evidence-based treatments such as CBT or BT [25,32] and generally lacked evidence supporting use and efficacy [21,23,25,27].”
BackgroundResultInfluentialWang, M., Chen, H., Yang, F., Xu, X., & Li, J. (2023) · Journal of Affective Disorders · doi:10.1016/j.jad.2023.06.057 · Cited by 25 · see citing works
The use of eHealth interventions among persons experiencing homelessness: A systematic review
Polillo, A., Gran-Ruaz, S., Sylvestre, J. C., & Kerman, N. (2021) · Digital Health · doi:10.1177/2055207620987066 · Cited by 25 · see citing works
Simon, L., Reimann, J., Steubl, L., Stach, M., Spiegelhalder, K., Sander, L., Baumeister, H., Meßner, E.-M., & Terhorst, Y. (2022) · Journal of Sleep Research · doi:10.1111/jsr.13642 · Cited by 24 · see citing works
Publication Trends in Psychotherapy: Bibliometric Analysis of the Past 5 Decades
Soares, E. E., Thrall, J., Stephens, T. N., Biglieri, R. R., Consoli, A. J., & Bunge, E. L. (2020) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.20190045 · Cited by 24 · see citing works
“Additionally, a growing number of publications have addressed the utilization of behavioral intervention technologies (52), such as smartphone applications (53, 54), artificial intelligence (55), virtual reality (56), chatbots (57, 58), and even video games utilizing CBT for depression (59) and anxiety (60).”
BackgroundMercurio, M., Larsen, M. E., Wisniewski, H., Henson, P., Lagan, S., & Torous, J. B. (2020) · Evidence-Based Mental Health · doi:10.1136/ebmental-2019-300137 · Cited by 23 · see citing works
Self-citationCan mental health apps be effective for depression, anxiety, and stress during a pandemic?
Longyear, R. L., & Kushlev, K. (2021) · Practice Innovations · doi:10.1037/pri0000142 · Cited by 21 · see citing works
Digital Health Literacy in Bipolar Disorder: International Web-Based Survey
Morton, E. E., Ho, K., Barnes, S. J., & Michalak, E. E. (2021) · JMIR Mental Health · doi:10.2196/29764 · Cited by 24 · see citing works
“…10,000 publicly available offerings [51], app store search algorithms lack transparency and may be influenced by paid advertising [52], turnover in apps is high [53], and app descriptions often make scientific claims regarding effectiveness despite lack of appropriate high-quality evidence [54,55].”
Kheirinejad, S., Visuri, A., Suryanarayana, S. A., & Hosio, S. J. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16586 · Cited by 23 · see citing works
Lin, X., Martinengo, L., Jabir, A. I., Ho, A. H. Y., Car, J., Atun, R. A., & Car, L. T. (2023) · Journal of Medical Internet Research · doi:10.2196/45984 · Cited by 24 · see citing works
“This could imply that some app developers might be using scientific terminology to attract users [63].”
Digital Technology to Enhance Clinical Care of Early Psychosis
Bell, I. H., & Álvarez‐Jiménez, M. (2019) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-019-00182-y · Cited by 23 · see citing works
Digital health & low-value care
O’Reilly-Jacob, M., Mohr, P. E., Ellen, M., Petersen, C. C., Sarkisian, C. A., Attipoe, S., & Rich, E. C. (2021) · Healthcare · doi:10.1016/j.hjdsi.2021.100533 · Cited by 22 · see citing works
Kang, R. M., & Reynolds, T. L. (2024) · International Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642178 · Cited by 27 · see citing works
“In a study conducted by Larsen and colleagues [38], researchers investigated app store description pages for mental health apps, fnding that 64% of apps included in their study claimed efcacy in reducing symptoms of mental illness or diagnosing a mental illness, and 44% of apps used “scientifc…”
BackgroundKadirvelu, B., Bel, T. B., Freccero, A., Simplicio, M. C. D., Nicholls, D., & Faisal, A. A. (2025) · Journal of Medical Internet Research · doi:10.2196/72501 · Cited by 30 · see citing works
“The proliferation of smartphones has enabled a new class of digital mental health interventions that leverage their unique data collection capabilities [12-14].”
- Authors
- Larsen, M. E., Huckvale, K., Nicholas, J., Torous, J., Birrell, L., Li, E., & Reda, B.
- Affiliations
- Black Dog Institute (AU)
- Northwestern University (US)
- Beth Israel Deaconess Medical Center (US)
- UNSW Sydney (AU)
- Corresponding author
- Larsen, Mark Erik
- Venue
- npj Digital Medicine
- Published
- 2019-03-22
- Volume
- 2
- Issue
- 1
- Pages
- 18-18
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC6550255
- OpenAlex
- W2925166053
- Cited by
- 334
- References
- 23
- Citations by year
- 2026: 16 · 2025: 53 · 2024: 42 · 2023: 48 · 2022: 47 · 2021: 60 · 2020: 54 · 2019: 14
- FWCI
- 49.43
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Keywords
- mental health apps, scientific language, quality of evidence, consumer choice, knowledge translation, depression, anxiety, schizophrenia, substance use, self-harm, Google Play Store, iTunes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
