From Research to Practice: Ten Lessons in Delivering Digital Mental Health Services
Open accessThe DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Abstract
There is a large body of research showing that psychological treatment can be effectively delivered via the internet, and Digital Mental Health Services (DMHS) are now delivering those interventions in routine care. However, not all attempts to translate these research outcomes into routine care have been successful. This paper draws on the experience of successful DMHS in Australia and Canada to describe ten lessons learned while establishing and delivering internet-delivered cognitive behavioural therapy (ICBT) and other mental health services as part of routine care. These lessons include learnings at four levels of analysis, including lessons learned working with (1) consumers, (2) therapists, (3) when operating DMHS, and (4) working within healthcare systems. Key themes include recognising that DMHS should provide not only treatment but also information and assessment services, that DMHS require robust systems for training and supervising therapists, that specialist skills are required to operate DMHS, and that the outcome data from DMHS can inform future mental health policy. We also confirm that operating such clinics is particularly challenging in the evolving funding, policy, and regulatory context, as well as increasing expectations from consumers about DMHS. Notwithstanding the difficulties of delivering DMHS, we conclude that the benefits of such services for the broader community significantly outweigh the challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“We intentionally avoided specific frameworks of reporting [36,37], because an aim of this paper was to describe the experiences of operating mature services, rather than just the implementation phase.”
MethodWhiteford, H. A., Degenhardt, L., Rehm, J., Baxter, A. J., Ferrari, A. J., Erskine, H. E., Charlson, F. J., Norman, R. E., Flaxman, A. D., Johns, N., et al. (2013) · The Lancet · doi:10.1016/s0140-6736(13)61611-6 · Cited by 6,006
The global burden of mental disorders: An update from the WHO World Mental Health (WMH) surveys
Kessler, R. C., Aguilar-Gaxiola, S., Alonso, J., Chatterji, S., Lee, S., Ormel, J., Üstün, T. B., & Wang, P. S. (2009) · Epidemiologia e Psichiatria Sociale · doi:10.1017/s1121189x00001421 · Cited by 1,895
Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013196 · Cited by 1,311
Self-citationAdvantages and limitations of Internet-based interventions for common mental disorders
Andersson, G., & Titov, N. (2014) · World Psychiatry · doi:10.1002/wps.20083 · Cited by 1,160
Self-citationAndersson, G., Cuijpers, P., Carlbring, P., Riper, H., & Hedman, E. (2014) · World Psychiatry · doi:10.1002/wps.20151 · Cited by 1,077 · cited 1×
“First, therapists new to DMHS often intellectually understand that DMHS treatments can result in similar outcomes to face-to-face treatment [10,59] but may initially expect DMHS treatments to produce poorer outcomes, a sentiment they may communicate to consumers [23].”
BackgroundSelf-citationThe RE-AIM framework: a systematic review of use over time
Gaglio, B., Shoup, J. A., & Glasgow, R. E. (2013) · American Journal of Public Health · doi:10.2105/ajph.2013.301299 · Cited by 1,029
Barriers to mental health treatment: results from the National Comorbidity Survey Replication
Mojtabai, R., Olfson, M., Sampson, N. A., Jin, R., Druss, B., Wang, P. S., Wells, K. B., Pincus, H. A., & Kessler, R. C. (2011) · Psychological Medicine · doi:10.1017/s0033291710002291 · Cited by 951
Guidelines for the practice of telepsychology.
Joint Task Force for the Development of Telepsychology Guidelines for Psychologists (2013) · American Psychologist · doi:10.1037/a0035001 · Cited by 535
Erbe, D., Eichert, H.-C., Riper, H., & Ebert, D. D. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6588 · Cited by 532
Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions
Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015) · Psychotherapy Research · doi:10.1080/10503307.2013.817696 · Cited by 494
Gilbody, S., Littlewood, E., Hewitt, C., Brierley, G., Tharmanathan, P., Araya, R., Barkham, M., Bower, P., Cooper, C., Gask, L., et al. (2015) · BMJ · doi:10.1136/bmj.h5627 · Cited by 409 · cited 1×
“Several attempts at implementation have either been unsuccessful [11] or were not found to have added value to existing face to face services [12],”
BackgroundConsensus statement on defining and measuring negative effects of internet interventions
Rozental, A., Andersson, G., Boettcher, J., Ebert, D. D., Cuijpers, P., Knaevelsrud, C., Ljótsson, B., Kaldo, V., Titov, N., & Carlbring, P. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.02.001 · Cited by 371 · cited 1×
“Fourth, we recommend that emerging DMHS measure and publish their outcomes, including disappointing and negative effects outcomes [84].”
BackgroundSelf-citationMohr, D. C., Weingardt, K. R., Reddy, M., & Schueller, S. M. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600541 · Cited by 365
Self-citationTitov, N., Dear, B. F., Johnston, L., Lorian, C., Zou, J., Wootton, B., Spence, J., McEvoy, P. M., & Rapee, R. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0062873 · Cited by 364 · cited 5×
“The OTU delivers several ICBT programs including an adaptation of the Wellbeing course developed at Macquarie University and used at the MindSpot Clinic [44,47,52].”
MethodInfluentialSelf-citationICBT in routine care: A descriptive analysis of successful clinics in five countries
Titov, N., Dear, B., Nielssen, O., Staples, L., Hadjistavropoulos, H., Nugent, M., Adlam, K., Nordgreen, T., Bruvik, K. H., Hovland, A., et al. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.07.006 · Cited by 353
Self-citationAndersson, G., Carlbring, P., Titov, N., & Lindefors, N. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719839381 · Cited by 324
Self-citationThe Use of Outcome Measures by Psychologists in Clinical Practice.
Hatfield, D. R., & Ogles, B. M. (2004) · Professional Psychology: Research and Practice · doi:10.1037/0735-7028.35.5.485 · Cited by 305
Dear, B. F., Staples, L. G., Terides, M. D., Karin, E., Zou, J., Johnston, L., Gandy, M., Fogliati, V. J., Wootton, B. M., McEvoy, P. M., et al. (2015) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2015.09.003 · Cited by 230 · cited 5×
“The OTU delivers several ICBT programs including an adaptation of the Wellbeing course developed at Macquarie University and used at the MindSpot Clinic [44,47,52].”
MethodInfluentialSelf-citationHedman, E., Ljótsson, B., Kaldo, V., Hesser, H., El Alaoui, S., Kraepelien, M., Andersson, E., Rück, C., Svanborg, C., Andersson, G., et al. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.10.023 · Cited by 224
Self-citationTitov, N., Dear, B. F., Staples, L. G., Terides, M. D., Karin, E., Sheehan, J., Johnston, L., Gandy, M., Fogliati, V. J., Wootton, B. M., et al. (2015) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2015.08.002 · Cited by 223
Self-citationVis, C., Mol, M., Kleiboer, A., Bührmann, L., Finch, T., Smit, J., & Riper, H. (2018) · JMIR Mental Health · doi:10.2196/mental.9769 · Cited by 183
Titov, N., Dear, B. F., Staples, L. G., Bennett-Levy, J., Klein, B., Rapee, R. M., Andersson, G., Purtell, C., Bezuidenhout, G., & Nielssen, O. B. (2017) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867416671598 · Cited by 183 · cited 3×
“MindSpot aims to improve access to evidence-based education, triage, assessment, referral, and treatment services throughout Australia to adults with symptoms of depression and anxiety [28,29].”
BackgroundSelf-citationDear, B. F., Titov, N., Perry, K. N., Johnston, L., Wootton, B. M., Terides, M. D., Rapee, R. M., & Hudson, J. L. (2013) · Pain · doi:10.1016/j.pain.2013.03.005 · Cited by 182 · cited 1×
“The clinic offers seven ICBT programs that have been validated in clinical trials, including transdiagnostic treatments designed to treat symptoms of anxiety and depression in several age groups [24,38–47] and disorder-specific treatments for obsessive compulsive disorder [48,49], post-traumatic stress disorder [50],…”
Self-citationNewby, J. M., Mackenzie, A., Williams, A. D., McIntyre, K., Watts, S., Wong, N., & Andrews, G. (2013) · Psychological Medicine · doi:10.1017/s0033291713000111 · Cited by 181
Titov, N., Dear, B. F., Staples, L. G., Bennett-Levy, J., Klein, B., Rapee, R. M., Shann, C., Richards, D., Andersson, G., Ritterband, L., et al. (2015) · Psychiatric Services · doi:10.1176/appi.ps.201400477 · Cited by 181
Self-citationChoi, I., Zou, J., Titov, N., Dear, B. F., Li, S., Johnston, L., Andrews, G., & Hunt, C. (2012) · Journal of Affective Disorders · doi:10.1016/j.jad.2011.11.003 · Cited by 180
Self-citationThe Effectiveness of Online Cognitive Behavioral Treatment in Routine Clinical Practice
Ruwaard, J., Lange, A., Schrieken, B., Dolan, C. V., & Emmelkamp, P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0040089 · Cited by 173 · cited 1×
“Notwithstanding the challenges, the successful use of ICBT as part of routine clinical care has been reported in Sweden [14–17], the Netherlands [18,19], Norway, Denmark [20,21], Canada [22–25], and Australia [26–30].”
BackgroundEstimating treatment rates for mental disorders in Australia
Whiteford, H. A., Buckingham, W. J., Harris, M. G., Burgess, P. M., Pirkis, J. E., Barendregt, J. J., & Hall, W. D. (2014) · Australian Health Review · doi:10.1071/ah13142 · Cited by 163
Hedman, E., Ljótsson, B., Rück, C., Bergström, J., Andersson, G., Kaldo, V., Jansson, L., Andersson, E., Andersson, E., Blom, K., et al. (2013) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12079 · Cited by 144
Self-citationKleiboer, A., Smit, J., Bosmans, J., Ruwaard, J., Andersson, G., Topooco, N., Berger, T., Krieger, T., Botella, C., Baños, R., et al. (2016) · Trials · doi:10.1186/s13063-016-1511-1 · Cited by 133
Self-citationRemote treatment of obsessive-compulsive disorder: A randomized controlled trial
Wootton, B. M., Dear, B. F., Johnston, L., Terides, M. D., & Titov, N. (2013) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2013.07.002 · Cited by 125 · cited 1×
“The clinic offers seven ICBT programs that have been validated in clinical trials, including transdiagnostic treatments designed to treat symptoms of anxiety and depression in several age groups [24,38–47] and disorder-specific treatments for obsessive compulsive disorder [48,49], post-traumatic stress disorder [50],…”
Self-citationFolker, A. P., Mathiasen, K., Lauridsen, S. M., Stenderup, E., Dozeman, E., & Folker, M. P. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.02.001 · Cited by 116 · cited 2×
“In addition to reports of outcomes from individual clinics, and reflecting the maturing state of the field, there is now an increasing number of studies describing barriers [20,23,31,32], guidelines for implementation [33,34], and comparisons of clinics across different countries [35].”
BackgroundDear, B. F., Staples, L. G., Terides, M. D., Fogliati, V. J., Sheehan, J., Johnston, L., Kayrouz, R., Dear, R., McEvoy, P. M., & Titov, N. (2016) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2016.05.004 · Cited by 114
Kenter, R. M. F., van de Ven, P. M., Cuijpers, P., Koole, G., Niamat, S., Gerrits, R. S., Willems, M., & van Straten, A. (2015) · Internet Interventions · doi:10.1016/j.invent.2015.01.001 · Cited by 105 · cited 1×
“Several attempts at implementation have either been unsuccessful [11] or were not found to have added value to existing face to face services [12],”
BackgroundHadjistavropoulos, H. D., Nugent, M. M., Alberts, N. M., Staples, L., Dear, B. F., & Titov, N. (2016) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2016.05.006 · Cited by 103 · cited 14×
“However, attempts to extend ICBT to routine care have produced mixed results.”
BackgroundMethodInfluentialSelf-citationEl Alaoui, S., Hedman, E., Kaldo, V., Hesser, H., Kraepelien, M., Andersson, E., Rück, C., Andersson, G., Ljótsson, B., & Lindefors, N. (2015) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0039198 · Cited by 99
Self-citationHadjistavropoulos, H. D., Nugent, M. M., Dirkse, D., & Pugh, N. (2017) · BMC Psychiatry · doi:10.1186/s12888-017-1496-7 · Cited by 94
Self-citationDear, B. F., Fogliati, V. J., Fogliati, R., Johnson, B., Boyle, O., Karin, E., Gandy, M., Kayrouz, R., Staples, L. G., & Titov, N. (2018) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867417738055 · Cited by 93
Self-citationTitov, N., Fogliati, V. J., Staples, L. G., Gandy, M., Johnston, L., Wootton, B., Nielssen, O., & Dear, B. F. (2016) · BJPsych Open · doi:10.1192/bjpo.bp.115.002139 · Cited by 93
Self-citationFogliati, V. J., Dear, B. F., Staples, L. G., Terides, M. D., Sheehan, J., Johnston, L., Kayrouz, R., Dear, R., McEvoy, P. M., & Titov, N. (2016) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2016.03.005 · Cited by 86
Titov, N., Dear, B. F., Johnston, L., McEvoy, P. M., Wootton, B., Terides, M. D., Gandy, M., Fogliati, V., Kayrouz, R., & Rapee, R. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089591 · Cited by 81
Self-citationHadjistavropoulos, H. D., Schneider, L. H., Edmonds, M., Karin, E., Nugent, M. N., Dirkse, D., Dear, B. F., & Titov, N. (2017) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2017.09.006 · Cited by 71 · cited 3×
“We encourage those seeking to start a DMHS to consider trialing different models of care to those currently used by existing DMHS, including testing different levels of therapist support [24,40,55] and testing care which combines both face-to-face and online delivery.”
BackgroundMethodSelf-citationHadjistavropoulos, H. D., Pugh, N. E., Hesser, H., & Andersson, G. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2014 · Cited by 61 · cited 1×
“New therapists also often find that their assumptions about mechanisms or facilitators of recovery are challenged, particularly when they learn that DMHS patients can develop very strong therapeutic alliances [60] and may obtain large clinical improvements, even after choosing not to have regular therapist contact [55…”
BackgroundSelf-citationHadjistavropoulos, H. D., Schneider, L. H., Klassen, K., Dear, B. F., & Titov, N. (2018) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2018.1457079 · Cited by 61 · cited 2×
“Since our clinics routinely administer PROMS and PREMS during and after treatment, we provide specific training for new therapists to increase their comfort and competence in using measures of outcome and experience [61].”
BackgroundMethodSelf-citationNielssen, O., Dear, B. F., Staples, L. G., Dear, R., Ryan, K., Purtell, C., & Titov, N. (2015) · BMC Psychiatry · doi:10.1186/s12888-015-0676-6 · Cited by 52 · cited 1×
“First, DMHS require robust procedures to define and effectively manage safety risks for people presenting with more severe and complex needs than seen in our clinical trials [30] and who often live in remote locations.”
Self-citationHedman, E., Ljótsson, B., Andersson, E., Rück, C., Andersson, G., & Lindefors, N. (2010) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2010.496460 · Cited by 49
Self-citationHadjistavropoulos, H. D., Schneider, L. H., Mehta, S., Karin, E., Dear, B. F., & Titov, N. (2019) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2019.02.002 · Cited by 46
Self-citationMathiasen, K., Riper, H., Andersen, T. E., & Roessler, K. K. (2018) · Journal of Medical Internet Research · doi:10.2196/10927 · Cited by 45
Urech, A., Krieger, T., Möseneder, L., Biaggi, A., Vincent, A., Poppe, C., Meyer, B., Riper, H., & Berger, T. (2019) · Psychotherapy Research · doi:10.1080/10503307.2018.1430910 · Cited by 43
Titov, N., Schofield, C., Staples, L., Dear, B. F., & Nielssen, O. (2019) · Australasian Psychiatry · doi:10.1177/1039856218789784 · Cited by 43 · cited 2×
“Our third lesson is that a diverse cross section of our communities access our services, including indigenous Australians and Canadians [56], people on low incomes, people living in rural and remote regions, and other groups who often under-utilise traditional health services.”
BackgroundSelf-citationHadjistavropoulos, H. D., Pugh, N. E., Hesser, H., & Andersson, G. (2016) · Behavior Therapy · doi:10.1016/j.beth.2015.10.006 · Cited by 42 · cited 1×
“For example, within the OTU, changes within recent years include (1) replacing disorder-specific ICBT programs with a transdiagnostic treatment program in light of the extensive comorbidity found among patients seeking services and the efficiency of delivery compared to disorder-specific programs [24], (2) ongoing tri…”
Self-citationHadjistavropoulos, H. D., Thompson, M. J., Klein, B., & Austin, D. W. (2012) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2011.645550 · Cited by 41
Self-citationHadjistavropoulos, H. D., Faller, Y. N., Klatt, A., Nugent, M. N., Dear, B. F., & Titov, N. (2018) · Community Mental Health Journal · doi:10.1007/s10597-018-0286-0 · Cited by 37 · cited 1×
“While most users access content online, many others download information and refer to it when they do not have internet access [54].”
BackgroundSelf-citationKayrouz, R., Dear, B. F., Kayrouz, B., Karin, E., Gandy, M., & Titov, N. (2018) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2018.1445124 · Cited by 33
Self-citationKayrouz, R., Dear, B. F., Karin, E., Fogliati, V. J., & Titov, N. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.06.002 · Cited by 32
Self-citationHadjistavropoulos, H. D., Gullickson, K. M., Schneider, L. H., Dear, B. F., & Titov, N. (2019) · Internet Interventions · doi:10.1016/j.invent.2019.100255 · Cited by 17
Self-citation
Cited by
“It happened to be the perfect thing”: experiences of generative AI chatbots for mental health
Siddals, S., Torous, J., & Coxon, A. (2024) · npj Mental Health Research · doi:10.1038/s44184-024-00097-4 · Cited by 241
Taylor, C. B., Fitzsimmons‐Craft, E. E., & Graham, A. K. (2020) · International Journal of Eating Disorders · doi:10.1002/eat.23300 · Cited by 183
“Although digital therapies are standard practice in many clinics around the world (Titov et al., 2019) and teletherapy has become routine practice in the United States' Veteran's Affairs Health Care sys-tem, on the whole, continued attention is needed to sustainably embed digital mental health in…”
BackgroundDavies, F., Shepherd, H. L., Beatty, L., Clark, B., Butow, P., & Shaw, J. (2020) · Journal of Medical Internet Research · doi:10.2196/17362 · Cited by 110
“Even successful online clinics do not see web-based therapy as suitable for all individuals [49], and patients also report a preference for face-to-face treatment over web based [53].”
BackgroundTitov, N., Dear, B. F., Nielssen, O., Wootton, B., Kayrouz, R., Karin, E., Genest, B., Bennett-Levy, J., Purtell, C., Bezuidenhout, G., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30224-7 · Cited by 108
Self-citationSchuster, R., Topooco, N., Keller, A., Radvogin, E., & Laireiter, A.-R. (2020) · Internet Interventions · doi:10.1016/j.invent.2020.100326 · Cited by 77
Metadata
- Authors
- Titov, N., Hadjistavropoulos, H. D., Nielssen, O., Mohr, D. C., Andersson, G., & Dear, B. F.
- Venue
- Journal of Clinical Medicine
- Publisher
- MDPI AG
- Published
- 2019-08-17
- Volume
- 8
- Issue
- 8
- Pages
- 1239
- ISSN
- 2077-0383 (ISSN-L)
- OpenAlex
- W2968620208
- Cited by
- 119
- References
- 57
- FWCI
- 16.61
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.