Stakeholder perspectives on evidence for digital mental health interventions: Implications for accreditation systems
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Background: Digital mental health interventions can be effective for treating mental health problems, but uptake by consumers and clinicians is not optimal. The lack of an accreditation pathway for digital mental health interventions is a barrier to their uptake among clinicians and consumers. However, there are a number of factors that may contribute to whether a digital intervention is suitable for recommendation to the public. The aim of this study was to identify the types of evidence that would support the accreditation of digital interventions. Method: An expert workshop was convened, including researcher, clinician, consumer (people with lived experience of a mental health condition) and policymaker representatives. Results: Existing methods for assessing the evidence for digital mental health interventions were discussed by the stakeholders present at the workshop. Empirical evidence from randomised controlled trials was identified as a key component for evaluating digital interventions. However, information on the safety of users, data security, user ratings, and fidelity to clinical guidelines, along with data from routine care including adherence, engagement and clinical outcomes, were also identified as important considerations when evaluating an intervention. There are considerable challenges in weighing the evidence for a digital mental health intervention. Conclusions: Empirical evidence should be the cornerstone of any accreditation system to identify appropriate digital mental health interventions. However, robust accreditation systems should also account for program and user safety, user engagement and experience, and fidelity to clinical treatment guidelines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In addition, 11 studies did not specify the analysis method used to analyze the data [33,41-50].”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [43]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Users’ engagement was facilitated by whether the intervention was endorsed by other users [43] or peers [211,241], their friends and family [97], or their current health care provider [152,161,242].”
From Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review (2021) · cites this paper as [43]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gulliver, A., Griffiths, K. M., & Christensen, H. (2010) · BMC Psychiatry · doi:10.1186/1471-244x-10-113 · Cited by 3,292 · cited 1×
“Clinician and consumer barriers to the use of these interventions include limited awareness of digital mental health interventions and their appropriateness for different mental health problems, preference for face-to-face care, lack of knowledge of the evidence base supporting their use, poorly integrated delivery pa…”
BackgroundMobile app rating scale: a new tool for assessing the quality of health mobile apps
Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015) · JMIR mHealth and uHealth · doi:10.2196/mhealth.3422 · Cited by 2,233
Self-citationEysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863
Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: A meta-analysis
SPEK, V., CUIJPERS, P., NYKLÍČEK, I., RIPER, H., KEYZER, J., & POP, V. (2007) · Psychological Medicine · doi:10.1017/s0033291706008944 · Cited by 1,586
Computer-based psychological treatments for depression: A systematic review and meta-analysis
Richards, D., & Richardson, T. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.02.004 · Cited by 1,442
Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013196 · Cited by 1,311
Self-citationEvaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847
Hollis, C., Falconer, C. J., Martin, J. L., Whittington, C., Stockton, S., Glazebrook, C., & Davies, E. B. (2017) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12663 · Cited by 846
Clark, D. M. (2011) · International Review of Psychiatry · doi:10.3109/09540261.2011.606803 · Cited by 699
Mobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality
Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4581 · Cited by 438 · cited 1×
“It should also be noted that user star ratings within general app stores have been shown to have limited correlation with measures of the clinical quality of apps.(28) Establishing independent panels of clinician and consumer users to provide a consistent rating process may overcome some of these challenges.”
Self-citationWaller, R., & Gilbody, S. (2009) · Psychological Medicine · doi:10.1017/s0033291708004224 · Cited by 393 · cited 1×
“Clinician and consumer barriers to the use of these interventions include limited awareness of digital mental health interventions and their appropriateness for different mental health problems, preference for face-to-face care, lack of knowledge of the evidence base supporting their use, poorly integrated delivery pa…”
BackgroundHuckvale, K., Prieto, J. T., Tilney, M., Benghozi, P.-J., & Car, J. (2015) · BMC Medicine · doi:10.1186/s12916-015-0444-y · Cited by 376 · cited 1×
“However, independently verifying these claims remains challenging.(21,22) There are other types of data that may indicate that a digital intervention is likely to be efficacious or effective, beyond RCTs.”
BackgroundICBT in routine care: A descriptive analysis of successful clinics in five countries
Titov, N., Dear, B., Nielssen, O., Staples, L., Hadjistavropoulos, H., Nugent, M., Adlam, K., Nordgreen, T., Bruvik, K. H., Hovland, A., et al. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.07.006 · Cited by 353
Self-citationMental health: there’s an app for that
Anthes, E. (2016) · Nature · doi:10.1038/532020a · Cited by 333 · cited 1×
“However, uptake by community members and clinicians for some evidence-based services has been limited, which may result from a failure of implementation into practice,(1,6) from limitations on the appropriateness of research evidence,(7) or from market saturation of untested interventions.(8,9) Digital mental health i…”
BackgroundHuckvale, K., Torous, J., & Larsen, M. E. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.2542 · Cited by 312 · cited 1×
“However, independently verifying these claims remains challenging.(21,22) There are other types of data that may indicate that a digital intervention is likely to be efficacious or effective, beyond RCTs.”
BackgroundSelf-citationA systematic assessment of smartphone tools for suicide prevention
Larsen, M. E., Nicholas, J., & Christensen, H. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152285 · Cited by 289 · cited 2×
“Fidelity to clinical guidelines or evidencebased practice could be assessed using an expert clinical review of the intervention, for example.(9) However, it was noted that a program may be entirely consistent with clinical guidelines but have poor outcomes, potentially due to low user engagement (e.”
BackgroundMethodChristensen, H., Griffiths, K. M., Korten, A. E., Brittliffe, K., & Groves, C. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.4.e46 · Cited by 240
Titov, N., Dear, B. F., Staples, L. G., Bennett-Levy, J., Klein, B., Rapee, R. M., Andersson, G., Purtell, C., Bezuidenhout, G., & Nielssen, O. B. (2017) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867416671598 · Cited by 183
Self-citationEffectiveness of Guided Internet-Based Cognitive Behavior Therapy in Regular Clinical Settings
Andersson, G., & Hedman, E. (2013) · Verhaltenstherapie · doi:10.1159/000354779 · Cited by 165 · cited 1×
“Programs that are delivered to the community as a clinical service may have considerable and detailed data on how specific types of users respond to programs over time.(23) Such services can continuously provide data on usage and clinical outcomes, can monitor user safety and can be used to identify appropriate clinic…”
BackgroundFDA Regulation of Mobile Medical Apps
Shuren, J., Patel, B., & Gottlieb, S. (2018) · JAMA · doi:10.1001/jama.2018.8832 · Cited by 155 · cited 1×
“Formal systems of accreditation for digital interventions are emerging, such as the UK Digital Assessment Questionnaire (DAQ)(15) and the US Food and Drug Administration’s procedures for approval of digital tools as medical devices.(16) However, the workshop was designed to identify the viewpoints of diverse stakehold…”
MethodEbert, D. D., Donkin, L., Andersson, G., Andrews, G., Berger, T., Carlbring, P., Rozenthal, A., Choi, I., Laferton, J. A. C., Johansson, R., et al. (2016) · Psychological Medicine · doi:10.1017/s0033291716001562 · Cited by 154
Self-citationAndrews, G., & Williams, A. D. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.05.006 · Cited by 85 · cited 1×
“Clinician and consumer barriers to the use of these interventions include limited awareness of digital mental health interventions and their appropriateness for different mental health problems, preference for face-to-face care, lack of knowledge of the evidence base supporting their use, poorly integrated delivery pa…”
BackgroundDeveloping a roadmap for the translation of e-mental health services for depression
Batterham, P. J., Sunderland, M., Calear, A. L., Davey, C. G., Christensen, H., Teesson, M., Kay-Lambkin, F., Andrews, G., Mitchell, P. B., Herrman, H., et al. (2015) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867415582054 · Cited by 77 · cited 1×
“Clinician and consumer barriers to the use of these interventions include limited awareness of digital mental health interventions and their appropriateness for different mental health problems, preference for face-to-face care, lack of knowledge of the evidence base supporting their use, poorly integrated delivery pa…”
BackgroundSelf-citationAn Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process
Zelmer, J., van Hoof, K., Notarianni, M., van Mierlo, T., Schellenberg, M., & Tannenbaum, C. (2018) · JMIR mHealth and uHealth · doi:10.2196/10016 · Cited by 62 · cited 1×
“However, there are broader considerations that may be important in assisting potential users to determine the most appropriate intervention.(14) Accreditation systems for digital interventions need to account for multiple factors in identifying interventions that are appropriate for public use and providing recommenda…”
BackgroundBaumel, A., & Muench, F. (2016) · JMIR Mental Health · doi:10.2196/mental.4563 · Cited by 51
Christensen, H., Murray, K., Calear, A. L., Bennett, K., Bennett, A., & Griffiths, K. M. (2010) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2010.tb03692.x · Cited by 44 · cited 1×
“based on randomised controlled trials and other empirical designs.(12) Loosely following an agreed agenda, lim-”
MethodSelf-citationBennion, M. R., Hardy, G. E., Moore, R. K., Kellett, S., & Millings, A. (2019) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2019-100027 · Cited by 15
Psyberguide: a path through the app jungle
Lipczynska, S. (2017) · Journal of Mental Health · doi:10.1080/09638237.2017.1417574 · Cited by 12
Cited by
Borghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069
“Users’ engagement was facilitated by whether the intervention was endorsed by other users [43] or peers [211,241], their friends and family [97], or their current health care provider [152,161,242].”
BackgroundRoss, S., Agin-Liebes, G., Lo, S., Zeifman, R. J., Ghazal, L., Benville, J., Franco Corso, S., Bjerre Real, C., Guss, J., Bossis, A., et al. (2021) · ACS Pharmacology & Translational Science · doi:10.1021/acsptsci.1c00020 · Cited by 139
Smartphone apps for the treatment of mental health conditions: status and considerations
Huckvale, K., Nicholas, J., Torous, J., & Larsen, M. E. (2020) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2020.04.008 · Cited by 135
Self-citationHuman-Computer Interaction in Digital Mental Health
Balcombe, L., & Leo, D. D. (2022) · Informatics · doi:10.3390/informatics9010014 · Cited by 188
“Digital mental health interventions are predominantly delivered by self-guided or clinician support (guided) approaches via online programs and apps (web-based and smartphone technologies) for evidence-based therapy to the user [30].”
BackgroundMitchell, L. M., Joshi, U., Patel, V., Lu, C., & Naslund, J. A. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.01.092 · Cited by 80
Metadata
- Authors
- Batterham, P. J., Calear, A. L., O’Dea, B., Larsen, M. E., J Kavanagh, D., Titov, N., March, S., Hickie, I., Teesson, M., Dear, B. F., Reynolds, J., Lowinger, J., Thornton, L., & Gorman, P.
- Affiliations
- Australian National University (AU)
- Black Dog Institute (AU)
- Centre for Children’s Health Research, Institute of Health & Biomedical Innovation, School of Psychology and Counselling, Queensland University of Technology, Brisbane, Australia
- Macquarie University (AU)
- University of Southern Queensland (AU)
- The University of Sydney (AU)
- Think Feel ACT, Melbourne, Australia
- Corresponding author
- Batterham, Philip J
- ORCID
- Venue
- DIGITAL HEALTH
- Publisher
- SAGE Publications
- Published
- 2019-01
- Volume
- 5
- Pages
- 2055207619878069-2055207619878069
- ISSN
- 2055-2076 (ISSN-L)
- PMC
- PMC6755623
- OpenAlex
- W2973694044
- Cited by
- 21
- References
- 28
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 3 · 2022: 2 · 2021: 8 · 2020: 1
- FWCI
- 3.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Digital Health (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- University of Southern Queensland ePrints (University of Southern Queensland) (acceptedVersion)
- QUT ePrints (Queensland University of Technology) (submittedVersion)
- QUT ePrints (Queensland University of Technology) (submittedVersion, cc-by-nc)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Health and Medical Research Council (1078407, 1083311, 1122544, 1128770, 1136259)
- Australian Department of Health
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Keywords
- digital mental health interventions, accreditation system, evidence requirements, stakeholder perspectives, randomized controlled trials, intervention safety, data security, user engagement, expert workshop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.