Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review
Open accessThe DOI matches its Crossref record; 102 of the 109 references it cites were found in a registry or an index; 232 of the 321 sources its claims cite were checked for support.
Abstract
Introduction: Online mental healthcare has gained significant attention due to its effectiveness, accessibility, and scalability in the management of mental health symptoms. Despite these advantages over traditional in-person formats, including higher availability and accessibility, issues with low treatment adherence and high dropout rates persist. Artificial intelligence (AI) technologies could help address these issues, through powerful predictive models, language analysis, and intelligent dialogue with users, however the study of these applications remains underexplored. The following mixed methods review aimed to supplement this gap by synthesizing the available evidence on the applications of AI in online mental healthcare. Method: We searched the following databases: MEDLINE, CINAHL, PsycINFO, EMBASE, and Cochrane. This review included peer-reviewed randomized controlled trials, observational studies, non-randomized experimental studies, and case studies that were selected using the PRISMA guidelines. Data regarding pre and post-intervention outcomes and AI applications were extracted and analyzed. A mixed-methods approach encompassing meta-analysis and network meta-analysis was used to analyze pre and post-intervention outcomes, including main effects, depression, anxiety, and study dropouts. We applied the Cochrane risk of bias tool and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) to assess the quality of the evidence. Results: Twenty-nine studies were included revealing a variety of AI applications including triage, psychotherapy delivery, treatment monitoring, therapy engagement support, identification of effective therapy features, and prediction of treatment response, dropout, and adherence. AI-delivered self-guided interventions demonstrated medium to large effects on managing mental health symptoms, with dropout rates comparable to non-AI interventions. The quality of the data was low to very low. Discussion: The review supported the use of AI in enhancing treatment response, adherence, and improvements in online mental healthcare. Nevertheless, given the low quality of the available evidence, this study highlighted the need for additional robust and high-powered studies in this emerging field. Systematic review registration: https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=443575, identifier CRD42023443575. Copyright © 2024 Gutierrez, Stephenson, Eadie, Asadpour and Alavi.
Overview
CoChat Summary
## TL;DR
This mixed-methods review found that AI-augmented online mental healthcare interventions were generally promising for reducing mental health symptoms and supporting engagement, but the evidence base was small and low quality. Across 29 studies, AI applications ranged from triage and psychotherapy delivery to predicting dropout and adherence, with self-guided AI interventions often performing well.
## Key findings
- The review included 29 studies and identified AI uses in triage, psychotherapy delivery, treatment monitoring, therapy engagement support, identification of effective therapy features, and prediction of treatment response, dropout, and adherence.
- For AI-augmented interventions, the meta-analysis found a significantly large effect size for overall mental health symptoms, a significant medium effect size for depression, and a significant small effect size for anxiety.
- In the network meta-analysis, AI-self-guided ACT was ranked best for reducing overall mental health symptoms, AI-self-guided eCBT was ranked best for reducing depression, and AI-guided eCBT was ranked best for reducing anxiety; however, the anxiety comparisons were not statistically significant.
- No included studies reported adverse events. For dropout, reported rates ranged from 0% to 61% for online or remote psychotherapy, 4% to 60% for psychoeducation, and 5% to 34% for no intervention; AI-self-guided eCBT had a 14% lesser risk of dropping out than psychoeducation in one significant comparison.
- User satisfaction was generally high: several studies reported AI tools were perceived as helpful and encouraging by around 60% to 90% of users, though around 30% or less found them unhelpful or bothersome.
## Methods
- Mixed-methods systematic review registered in PROSPERO (CRD42023443575) and conducted according to PRISMA and PRISMA-NMA guidance.
- Searched MEDLINE, CINAHL, PsycINFO, EMBASE, and Cochrane for English-language studies published up to October 2023; included RCTs, observational studies, non-randomized experimental studies, and case studies.
- Extracted pre/post outcomes and AI applications; used meta-analysis and network meta-analysis for symptom and dropout outcomes, plus narrative synthesis for machine-learning prediction studies; assessed quality with the Cochrane risk of bias tool and GRADE.
## Limitations
- The paper states that the evidence was low to very low quality under GRADE, with many studies showing high or unclear risk of bias, especially allegiance bias (n=14) and selective reporting (n=8).
- The authors note a major limitation was the scarcity of studies: many interventions/comparators were represented by one study only, which increased imprecision, indirectness, and bias; several NMA comparisons were therefore based on indirect evidence.
- Evident from the text: the review relied largely on Western, English-language studies and mostly female samples, which the authors say may limit generalizability; also, adverse events may be underreported in this literature, so “none reported” does not necessarily mean none occurred.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
82 Valid · 9 Not valid · 12 Not checked · 73 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“The studies that ranked high for selective reporting bias tended to report only user satisfaction data and treatment adherence, despite their methods considering the collection and assessment of psychiatric scale data and other effectiveness outcomes (29, 76, 79, 85–88, 92).”
§ Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review > Results > Risk of bias within studies
What the sources say
- Disputedcites [29] Flygare, 2020, Predictors of remission from body dysmorphic disorder after internet-delivered cognitive behavior therapy: A machine learning approach, BMC Psychiatry sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
- Disputedcites [76] Fulmer, 2018, Using psychological artificial intelligence (tess) to relieve symptoms of depression and anxiety: Randomized controlled trial, JMIR Ment Health sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 1 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
- Disputedcites [79] Morrison, 2017, The effect of timing and frequency of push notifications on usage of a smartphone-based stress management intervention: An exploratory trial, PloS One sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
- Not checked — no full text of the sourcecites [85] Ewbank, 2021, Understanding the relationship between patient language and outcomes in internet-enabled cognitive behavioural therapy: A deep learning approach to automatic coding of session transcripts, Psychother Res source
- Partly supportedcites [86] Ewbank, 2020, Quantifying the association between psychotherapy content and clinical outcomes using deep learning, JAMA Psychiatry sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
- Disputedcites [87] Kim, 2021, Machine learning analysis to identify digital behavioral phenotypes for engagement and health outcome efficacy of an mHealth intervention for obesity: Randomized controlled trial, J Med Internet Res sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
- Disputedcites [88] Lenhard, 2018, Prediction of outcome in internet-delivered cognitive behaviour therapy for paediatric obsessive-compulsive disorder: A machine learning approach, Int J Methods Psychiatr Res sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
- Disputedcites [92] Rodrigo, 2021, Exploratory data mining techniques (decision tree models) for examining the impact of internet-based cognitive behavioral therapy for tinnitus: Machine learning approach, J Med Internet Res sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
plainstates an associationn = 8- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Additionally, studies have shown that despite some concerns over the strength of the therapeutic relationship, online mental healthcare has similar effectiveness as in-person options for the management of mental health conditions (11, 24).”
§ Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review > Introduction
What the sources say
- Not checked — no full text of the sourcecites [11] Bennett, 2020, eHealth to redress psychotherapy access barriers both new and old: A review of reviews and meta-analyses, J Psychother Integr source
- Partly supportedcites [24] McDonald, 2020, Online psychotherapy: trailblazing digital healthcare, BJPsych Bull sourceExperimental — accuracy not yet measured
The ‘Evidence base for online therapies’ section reports improvements with online therapies, particularly CBT, but its comparisons are mainly with treatment-as-usual or waiting-list groups. The ‘Financial costs’ section says many online programmes have been effective; it compares costs, not clinical effectiveness, with face-to-face therapy. The paper therefore supports online treatment's effectiveness but not the claim that it is similarly effective to in-person care. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, the study of the applications of AI in mental healthcare remains in its infancy (6, 20).”
- Also stated in: § Introduction · plain
“In contrast, the application of AI in mental healthcare remains in its infancy (6, 20).”
§ Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review > Introduction
What the sources say
- Not checked — no full text of the sourcecites [6] Boucher, 2021, Artificially intelligent chatbots in digital mental health interventions: a review, Expert Rev Med Devices source
- Supportedcites [20] Graham, 2019, Artificial intelligence for mental health and mental illnesses: an overview, Curr Psychiatry Rep source
The Abstract’s Recent findings section says most of the 28 reviewed studies are early proof-of-concept work. The Results section, under Limitations of AI and mental health studies, also identifies gaps in clinical validation and readiness for patient care. Together, these support the characterization of research on AI applications in mental healthcare as being in its infancy. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“AI technology can help address these issues by supporting a versatile, scalable, and powerful approach that can analyze and respond to large amounts of data and adapt to individual healthcare needs (6, 17–23).”
§ Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review > Introduction
What the sources say
- Not checked — no full text of the sourcecites [6] Boucher, 2021, Artificially intelligent chatbots in digital mental health interventions: a review, Expert Rev Med Devices source
- Partly supportedcites [17] Abd-alrazaq, 2019, An overview of the features of chatbots in mental health: A scoping review, Int J Med Inform sourceExperimental — accuracy not yet measured
The Introduction says chatbots may increase access to mental-health interventions, and §4.3.1 says AI chatbots can respond to complex questions and learn from gathered data. That section also says web-based chatbots could reach many users. The paper does not establish that AI chatbots can process and respond to large volumes of data or adapt care to each individual's needs. (LLM, conf=0.85)
- Partly supportedcites [18] Barnett, 2021, Enacting ‘more-than-human’ care: Clients’ and counsellors’ views on the multiple affordances of chatbots in alcohol and other drug counselling, Int J Drug Policy sourceExperimental — accuracy not yet measured
The Introduction says chatbots may bring mental health services to larger audiences, and the Findings describe their potential to tailor alcohol and other drug care to a client’s needs. The paper also describes natural language processing as analysing and responding to human language, but it does not establish the claimed ability to handle large amounts of data; its account of tailored care is a possibility, not a demonstrated capability. Page numbers were unavailable in our copy. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [19] Fonseka, 2019, The utility of artificial intelligence in suicide risk prediction and the management of suicidal behaviors, Aust New Z J Psychiatry source
- Partly supportedcites [20] Graham, 2019, Artificial intelligence for mental health and mental illnesses: an overview, Curr Psychiatry Rep sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says AI can draw meaning from large, varied data sources, support remote therapeutic sessions, and potentially tailor treatments to individuals. It does not establish that this approach is scalable or specifically resolves the citing document’s online mental-healthcare issues. (LLM, conf=0.83)
- Partly supportedcites [21] Bennett, 2013, Artificial intelligence framework for simulating clinical decision-making: A Markov decision process approach, Artif Intell Med sourceExperimental — accuracy not yet measured
Sections 1.1 and 4.1 describe using AI to learn from clinical data and adjust decisions as information changes; Section 2.1 reports evaluation with mental-health patient records. Section 1.2 identifies telemedicine as a potential role, but the paper does not establish that this approach addresses online mental healthcare issues or demonstrate its scalability. (LLM, conf=0.84)
- Partly supportedcites [22] Cresswell, 2020, Investigating the use of data-driven artificial intelligence in computerised decision support systems for health and social care: A systematic review, Health Inform J sourceExperimental — accuracy not yet measured
The Background says data-driven decision-support systems analyze large, complex datasets, and the Descriptive synthesis describes personalized warfarin-dosing recommendations. The paper does not establish that these systems address online mental healthcare issues or that they are scalable; its Policy recommendations caution that evidence of effectiveness remains limited. (LLM, conf=0.86)
- Partly supportedcites [23] Lysaght, 2019, AI-assisted decision-making in healthcare, Asian Bioeth Rev sourceExperimental — accuracy not yet measured
The Background describes AI systems that analyze large datasets and use individual patients’ clinical data to support diagnoses and predict treatment outcomes. It does not address online mental healthcare or establish that this approach is scalable in that setting; those parts of the claim go beyond the paper. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“If carefully implemented, ML can produce results with a precision that can rival or surpass seasoned clinicians (20).”
§ Examining the role of AI technology in online mental healthcare: opportunities, challenges, and implications, a mixed-methods review > Introduction
What the sources say
- Supportedcites [20] Graham, 2019, Artificial intelligence for mental health and mental illnesses: an overview, Curr Psychiatry Rep source
The “AI in healthcare” section says AI algorithms can perform as well as or better than experienced clinicians when evaluating medical images. This supports the claim that such performance is possible, though the paper’s example concerns image evaluation rather than all clinical tasks. (LLM, conf=0.88)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
84 more claims · 38 with their cited sources checked
Create a free account to see all 176 claims and every source verdict.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 65
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Hutton, B., Salanti, G., Caldwell, D. M., Chaimani, A., Schmid, C. H., Cameron, C., Ioannidis, J. P. A., Straus, S., Thorlund, K., Jansen, J. P., et al. (2015) · Annals of Internal Medicine · doi:10.7326/m14-2385 · Cited by 8,269
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793
How to perform a meta-analysis with R: a practical tutorial
Balduzzi, S., Rücker, G., & Schwarzer, G. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2019-300117 · Cited by 5,827
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064 · cited 1×
“These techniques show the number of hypothetical null results that would be required to make the results of the analysis insigni fi cant (56).”
BackgroundSalanti, G. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1037 · Cited by 1,729
How Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294
Chatbots: History, technology, and applications
Adamopoulou, E., & Moussiades, L. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100006 · Cited by 1,090 · cited 1×
“This limitation can be frustrating for users if they feel that the AI service does not understand them or their needs, making the interaction feel less human-like impacting treatment adherence, and may impact the prediction abilities and accuracy of large language models (6, 17, 18, 99).”
BackgroundEvaluation of networks of randomized trials
Salanti, G., Higgins, J. P., Ades, A., & Ioannidis, J. P. (2008) · Statistical Methods in Medical Research · doi:10.1177/0962280207080643 · Cited by 1,078
Inkster, B., Sarda, S., & Subramanian, V. (2018) · doi:10.2196/preprints.12106 · Cited by 1,063
Artificial intelligence for mental health and mental illnesses: an overview
Graham, S., Depp, C., Lee, E. E., Nebeker, C., Tu, X., Kim, H.-C., & Jeste, D. V. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1094-0 · Cited by 970
Torous, J., Jän Myrick, K., Rauseo-Ricupero, N., & Firth, J. (2020) · JMIR Mental Health · doi:10.2196/18848 · Cited by 946
Network meta-analysis: an introduction for clinicians
Rouse, B., Chaimani, A., & Li, T. (2017) · Internal and Emergency Medicine · doi:10.1007/s11739-016-1583-7 · Cited by 875 · cited 2×
“Moreover as mentioned by other reviews in the fi eld, several of the available studies lack an appropriate comparator or control group which can directly impact the observed effect of the AI-augmented interventions and tools reported in the literature (6, 17 – 23, 52 – 54, 57, 58, 67, 72).”
MethodInfluentialMalhi, G. S., Bassett, D., Boyce, P., Bryant, R., Fitzgerald, P. B., Fritz, K., Hopwood, M., Lyndon, B., Mulder, R., Murray, G., et al. (2015) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867415617657 · Cited by 842
Fulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018) · doi:10.2196/preprints.9782 · Cited by 715
Approaches to interpreting and choosing the best treatments in network meta-analyses
Mbuagbaw, L., Rochwerg, B., Jaeschke, R., Heels-Andsell, D., Alhazzani, W., Thabane, L., & Guyatt, G. H. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0473-z · Cited by 693
Jansen, J. P., & Naci, H. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-159 · Cited by 656
Harnessing context sensing to develop a mobile intervention for depression
Burns, M. N., Begale, M., Duffecy, J., Gergle, D., Karr, C. J., Giangrande, E., & Mohr, D. C. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1838 · Cited by 646 · cited 8×
“They also reported that a minority of users, around 30% or less, thought that the tool did not understand them or found their interactions unhelpful or bothersome (73, 74, 76 – 78, 80 – 82, 84, 90).”
BackgroundMethodResultInfluentialAn overview of the features of chatbots in mental health: A scoping review
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103978 · Cited by 605
Multivariate statistical machine learning methods for genomic prediction
Montesinos López, O. A., Montesinos López, A., & Crossa, J. (2022) · doi:10.1007/978-3-030-89010-0 · Cited by 516
Confidence Interval or P-Value? Part 4 of a Series on Evaluation of Scientific Publications
Prel, J.-B. D., Hommel, G., Röhrig, B., & Blettner, M. (2009) · Deutsches Ärzteblatt international · doi:10.3238/arztebl.2009.0335 · Cited by 460
Artificially intelligent chatbots in digital mental health interventions: a review
Boucher, E. M., Harake, N. R., Ward, H. E., Stoeckl, S. E., Vargas, J., Minkel, J., Parks, A. C., & Zilca, R. (2021) · Expert Review of Medical Devices · doi:10.1080/17434440.2021.2013200 · Cited by 440
Owen, R. K., Bradbury, N., Xin, Y., Cooper, N., & Sutton, A. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1373 · Cited by 427 · cited 2×
“For both meta-analysis and NMA, we employed a random-effects model to calculate the SMD or RR for relevant outcomes in the included studies, and heterogeneity using t 2 and I 2 with the meta package in R version 3.5.3, computed using RStudio and the NMA tool, Meta Insight (59, 61, 68 – 70).”
MethodInfluentialThe workforce for health in a globalized context – global shortages and international migration
Aluttis, C., Bishaw, T., & Frank, M. W. (2014) · Global Health Action · doi:10.3402/gha.v7.23611 · Cited by 423
Bennett, C. C., & Hauser, K. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.12.003 · Cited by 378
Mental health services in Canada: Barriers and cost-effective solutions to increase access
Moroz, N., Moroz, I., & D’Angelo, M. S. (2020) · Healthcare Management Forum · doi:10.1177/0840470420933911 · Cited by 352
Lysaght, T., Lim, H. Y., Xafis, V., & Ngiam, K. Y. (2019) · Asian Bioethics Review · doi:10.1007/s41649-019-00096-0 · Cited by 332 · cited 10×
“…risk assessment and risk prediction, identi fi cation of mental illness predictors, mental health monitoring, psychoeducation and psychotherapy delivery, therapist training, mental healthcare personalization, mental health triage and decision-making, and promoting treatment engagement (6, 17 – 23).”
BackgroundMethodResultInfluentialHow to read a funnel plot in a meta-analysis
Sedgwick, P., & Marston, L. (2015) · BMJ · doi:10.1136/bmj.h4718 · Cited by 318 · cited 2×
“Analysis of funnel plots (Supplementary Material 8) and Rosenthal ’ s fail-safe N (Table 2) showed that most reported outcomes presented a low degree of publication bias (p 0.05), except for anxiety and dropouts in the NMA (55).”
BackgroundMethodInfluentialDOME: recommendations for supervised machine learning validation in biology
Walsh, I., Fishman, D., Garcia-Gasulla, D., Titma, T., Pollastri, G., ELIXIR Machine Learning Focus Group, Capriotti, E., Casadio, R., Capella-Gutierrez, S., Cirillo, D., et al. (2021) · Nature Methods · doi:10.1038/s41592-021-01205-4 · Cited by 282 · cited 7×
“…followed the recent research guidelines and standards for ML studies, which recommend that these studies report on the adequacy of the data for the intended outcomes, model training and fi ne-tuning, features analyzed, validation, interpretability, and code and data availability (34 – 37).”
BackgroundMethodInfluentialMusiat, P., Goldstone, P., & Tarrier, N. (2014) · BMC Psychiatry · doi:10.1186/1471-244x-14-109 · Cited by 273
Proudfoot, J., Clarke, J., Birch, M.-R., Whitton, A. E., Parker, G., Manicavasagar, V., Harrison, V., Christensen, H., & Hadzi-Pavlovic, D. (2013) · BMC Psychiatry · doi:10.1186/1471-244x-13-312 · Cited by 264
Artificial intelligence and the future of psychiatry: Insights from a global physician survey
Doraiswamy, P. M., Blease, C., & Bodner, K. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.101753 · Cited by 253
Emotion dysregulation in adults with attention deficit hyperactivity disorder: a meta-analysis
Beheshti, A., Chavanon, M.-L., & Christiansen, H. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-2442-7 · Cited by 247
Klos, M. C., Escoredo, M., Joerin, A., Lemos, V. N., Rauws, M., & Bunge, E. L. (2021) · JMIR Formative Research · doi:10.2196/20678 · Cited by 200
Musiat, P., & Tarrier, N. (2014) · Psychological Medicine · doi:10.1017/s0033291714000245 · Cited by 188
He, Y., Yang, L., Zhu, X., Wu, B., Zhang, S., Qian, C., & Tian, T. (2022) · Journal of Medical Internet Research · doi:10.2196/40719 · Cited by 178
Morrison, L. G., Hargood, C., Pejovic, V., Geraghty, A. W. A., Lloyd, S., Goodman, N., Michaelides, D. T., Weston, A., Musolesi, M., Weal, M. J., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0169162 · Cited by 162
Castellini, G., Bruschettini, M., Gianola, S., Gluud, C., & Moja, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0770-1 · Cited by 161 · cited 3×
“Moreover as mentioned by other reviews in the fi eld, several of the available studies lack an appropriate comparator or control group which can directly impact the observed effect of the AI-augmented interventions and tools reported in the literature (6, 17 – 23, 52 – 54, 57, 58, 67, 72).”
MethodResultInfluentialBrinker, T. J., Hekler, A., Hauschild, A., Berking, C., Schilling, B., Enk, A. H., Haferkamp, S., Karoglan, A., von Kalle, C., Weichenthal, M., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2018.12.016 · Cited by 157
Månsson, K. N. T., Frick, A., Boraxbekk, C.-J., Marquand, A. F., Williams, S. C. R., Carlbring, P., Andersson, G., & Furmark, T. (2015) · Translational Psychiatry · doi:10.1038/tp.2015.22 · Cited by 154
Osenbach, J. E., O'Brien, K. M., Mishkind, M., & Smolenski, D. J. (2013) · Depression and Anxiety · doi:10.1002/da.22165 · Cited by 144 · cited 1×
“The various AI-augmented applications presented in this review have achieved encouraging positive mental health outcomes, while leveraging and supporting the bene fi ts of online platforms, including scalability, versatility, cost-effectiveness, and personalization (8, 9, 95 – 97).”
BackgroundSztein, D. M., Koransky, C. E., Fegan, L., & Himelhoch, S. (2018) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x17717402 · Cited by 135
Sarri, G., Patorno, E., Yuan, H., Guo, J., Bennett, D., Wen, X., Zullo, A. R., Largent, J. A., Panaccio, M. P., Gokhale, M., et al. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111493 · Cited by 135
Can't surf, won't surf: The digital divide in mental health
Ennis, L., Rose, D., Denis, M., Pandit, N., & Wykes, T. (2012) · Journal of Mental Health · doi:10.3109/09638237.2012.689437 · Cited by 127 · cited 1×
“Additionally, it is important to acknowledge that the implementation and use of these novel and promising interventions may be restricted to those with a reliable level of technology literacy, and access to the internet and internet-enabled devices (95, 101).”
BackgroundRecommendations and future directions for supervised machine learning in psychiatry
Cearns, M., Hahn, T., & Baune, B. T. (2019) · Translational Psychiatry · doi:10.1038/s41398-019-0607-2 · Cited by 127
Fonseka, T. M., Bhat, V., & Kennedy, S. H. (2019) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867419864428 · Cited by 126
Internet-delivered cognitive behavioural therapy in the treatment of psychiatric illness
Gratzer, D., & Khalid-Khan, F. (2016) · Canadian Medical Association Journal · doi:10.1503/cmaj.150007 · Cited by 112
Chien, I., Enrique, A., Palacios, J., Regan, T., Keegan, D., Carter, D., Tschiatschek, S., Nori, A., Thieme, A., Richards, D., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.10791 · Cited by 111
Quantifying the association between psychotherapy content and clinical outcomes using deep learning
Ewbank, M. P., Cummins, R., Tablan, V., Bateup, S., Catarino, A., Martin, A. J., & Blackwell, A. D. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.2664 · Cited by 108 · cited 6×
“(85, 86, 91)] focused on the content of the therapy program and the utterances of the participants and found several factors that could promote and predict higher effectiveness and engagement.”
BackgroundResultInfluentialForsell, E., Jernelöv, S., Blom, K., Kraepelien, M., Svanborg, C., Andersson, G., Lindefors, N., & Kaldo, V. (2019) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.18060699 · Cited by 100
Eligibility screening in evidence synthesis of environmental management topics
Frampton, G. K., Livoreil, B., & Petrokofsky, G. (2017) · Environmental Evidence · doi:10.1186/s13750-017-0102-2 · Cited by 95
Hidalgo-Mazzei, D., Mateu, A., Reinares, M., Matic, A., Vieta, E., & Colom, F. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.08.005 · Cited by 92
Cresswell, K., Callaghan, M., Khan, S., Sheikh, Z., Mozaffar, H., & Sheikh, A. (2020) · Health Informatics Journal · doi:10.1177/1460458219900452 · Cited by 87
Bennett, C. B., Ruggero, C. J., Sever, A. C., & Yanouri, L. (2020) · Journal of Psychotherapy Integration · doi:10.1037/int0000217 · Cited by 87 · cited 2×
“Online mental healthcare presents several advantages compared to its in-person counterpart, notably additional privacy, and the ability to access healthcare anywhere with an internet connection (11, 24).”
BackgroundOnline and mobile technologies for self-management in bipolar disorder: A systematic review.
Gliddon, E., Barnes, S. J., Murray, G., & Michalak, E. E. (2017) · Psychiatric Rehabilitation Journal · doi:10.1037/prj0000270 · Cited by 84
Lenhard, F., Sauer, S., Andersson, E., Månsson, K. N., Mataix‐Cols, D., Rück, C., & Serlachius, E. (2018) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1576 · Cited by 84
Danieli, M., Ciulli, T., Mousavi, S. M., Silvestri, G., Barbato, S., Di Natale, L., & Riccardi, G. (2022) · JMIR Mental Health · doi:10.2196/38067 · Cited by 84
World Health Organization's Comprehensive Mental Health Action Plan 2013–2020
Saxena, S., & Setoya, Y. (2014) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.12207 · Cited by 81
Telepsychiatry and e-Mental Health Services: Potential for improving Access to Mental Health Care
McGinty, K. L., Saeed, S. A., Simmons, S. C., & Yildirim, Y. (2006) · Psychiatric Quarterly · doi:10.1007/s11126-006-9019-6 · Cited by 81 · cited 3×
“Additionally, it is important to acknowledge that the implementation and use of these novel and promising interventions may be restricted to those with a reliable level of technology literacy, and access to the internet and internet-enabled devices (95, 101).”
BackgroundPearson, R., Pisner, D., Meyer, B., Shumake, J., & Beevers, C. G. (2019) · Psychological Medicine · doi:10.1017/s003329171800315x · Cited by 81 · cited 6×
“Eight studies reported on the ethnicity of the participants (30, 32, 41, 74, 76, 77, 80, 83), and in most of the studies between 31% and 91% of the participants identi fi ed as White.”
BackgroundResultInfluentialState of the e-mental health field in Australia: Where are we now?
Christensen, H., & Petrie, K. (2013) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867412471439 · Cited by 67
Johansson, M., Sinadinovic, K., Gajecki, M., Lindner, P., Berman, A. H., Hermansson, U., & Andréasson, S. (2020) · Addiction · doi:10.1111/add.15270 · Cited by 66
Anthony, C. A., Rojas, E. O., Keffala, V., Glass, N. A., Shah, A. S., Miller, B. J., Hogue, M., Willey, M. C., Karam, M., & Marsh, J. L. (2020) · doi:10.2196/preprints.17750 · Cited by 64 · cited 18×
“For RCTs reporting treatment dropouts, we employed an NMA approach as detailed above to obtain a deeper understanding of the attrition rates among the identi fi ed AI-augmented interventions.”
InfluentialWallert, J., Gustafson, E., Held, C., Madison, G., Norlund, F., von Essen, L., & Olsson, E. M. G. (2018) · doi:10.2196/preprints.10754 · Cited by 63
Intelligent Computer-Based Assessment and Psychotherapy: An Expert System for Sexual Dysfunction
BINIK, Y. M., SERVAN-SCHREIBER, D., FREIWALD, S., & HALL, K. S. K. (1988) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-198807000-00001 · Cited by 53 · cited 4×
“Binik et al. (1988) reported that users of their AI-delivered virtual sexual dysfunction therapy tool (Sexpert) appeared comfortable discussing issues with their sexuality and other intimate topics with the tool (82).”
BackgroundResultInfluentialFlygare, O., Enander, J., Andersson, E., Ljótsson, B., Ivanov, V. Z., Mataix-Cols, D., & Rück, C. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02655-4 · Cited by 51
Barnett, A., Savic, M., Pienaar, K., Carter, A., Warren, N., Sandral, E., Manning, V., & Lubman, D. I. (2021) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2020.102910 · Cited by 50
Lagan, S., Ramakrishnan, A., Lamont, E., Ramakrishnan, A., Frye, M., & Torous, J. (2020) · International Journal of Bipolar Disorders · doi:10.1186/s40345-020-00202-4 · Cited by 50
Will Artificial Intelligence Replace the Human Echocardiographer?: Clinical Considerations
Sengupta, P. P., & Adjeroh, D. A. (2018) · Circulation · doi:10.1161/circulationaha.118.037095 · Cited by 50
Wallert, J., Boberg, J., Kaldo, V., Mataix-Cols, D., Flygare, O., Crowley, J. J., Halvorsen, M., Ben Abdesslem, F., Boman, M., Andersson, E., et al. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-02133-3 · Cited by 48
Meheli, S., Sinha, C., & Kadaba, M. (2022) · JMIR Human Factors · doi:10.2196/35671 · Cited by 46
Online psychotherapy: trailblazing digital healthcare
McDonald, A., Eccles, J. A., Fallahkhair, S., & Critchley, H. D. (2020) · BJPsych Bulletin · doi:10.1192/bjb.2019.66 · Cited by 45 · cited 8×
“Especially in the case of fully self-guided online psychotherapy, the lack of monitoring, risk assessment and personalization may place the patient at an increased risk of dropping out from the treatment or experiencing an exacerbation of psychiatric symptoms (20, 24).”
BackgroundInfluentialSundström, C., Eék, N., Kraepelien, M., Fahlke, C., Gajecki, M., Jakobson, M., Beckman, M., Kaldo, V., & Berman, A. H. (2020) · Addiction · doi:10.1111/add.14871 · Cited by 42
Kim, M., Yang, J., Ahn, W.-Y., & Choi, H. J. (2021) · Journal of Medical Internet Research · doi:10.2196/27218 · Cited by 41
Ewbank, M. P., Cummins, R., Tablan, V., Catarino, A., Buchholz, S., & Blackwell, A. D. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1788740 · Cited by 39
Jesus‐Nunes, A. P., Leal, G. C.-G., Correia‐Melo, F. S., Vieira, F., Mello, R. P., Caliman‐Fontes, A. T., Echegaray, M. V. F., Marback, R. F., Guerreiro‐Costa, L. N. F., Souza‐Marques, B., et al. (2022) · Human Psychopharmacology Clinical and Experimental · doi:10.1002/hup.2836 · Cited by 39 · cited 4×
“Rodrigo et al. (2022) applied an ANN and machine learning approach and determined that higher education level, older age, employment status (i.e., no or fewer work restrictions), baseline tinnitus severity, and lower insomnia scores were predictors of treatment response with 78% accuracy.”
MethodInfluentialPredicting early dropout in online versus face-to-face guided self-help: A machine learning approach
Gonzalez Salas Duhne, P., Delgadillo, J., & Lutz, W. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2022.104200 · Cited by 38
Alavi, N., Moghimi, E., Stephenson, C., Gutierrez, G., Jagayat, J., Kumar, A., Shao, Y., Miller, S., Yee, C. S., Stefatos, A., et al. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1113956 · Cited by 36
Online CBT Is Effective in Overcoming Cultural and Language Barriers in Patients With Depression
ALAVI, N., HIRJI, A., SUTTON, C., & NAEEM, F. (2016) · Journal of Psychiatric Practice · doi:10.1097/pra.0000000000000119 · Cited by 34
Self-citationRodrigo, H., Beukes, E. W., Andersson, G., & Manchaiah, V. (2021) · doi:10.2196/preprints.28999 · Cited by 33
Bremer, V., Chow, P. I., Funk, B., Thorndike, F. P., & Ritterband, L. M. (2020) · doi:10.2196/preprints.17738 · Cited by 29
Alavi, N., Yang, M., Stephenson, C., Nikjoo, N., Malakouti, N., Layzell, G., Jagayat, J., Shirazi, A., Groll, D., Omrani, M., et al. (2020) · JMIR Research Protocols · doi:10.2196/24913 · Cited by 27
Self-citationOnline Cognitive Behavioral Therapy: An e-Mental Health Approach to Depression and Anxiety
Alavi, N., & Omrani, M. (2019) · doi:10.1007/978-3-319-99151-1 · Cited by 25
Self-citationAlavi, N., Stephenson, C., & Rivera, M. (2021) · JMIR Mental Health · doi:10.2196/27308 · Cited by 24
Self-citationFrías, Á., Palma, C., Salvador, A., Aluco, E., Navarro, S., Farriols, N., Aliaga, F., Solves, L., & Antón, M. (2021) · Australasian Psychiatry · doi:10.1177/1039856220924321 · Cited by 18
Mylläri, S., Saarni, S. E., Ritola, V., Joffe, G., Stenberg, J., Solbakken, O. A., Czajkowski, N. O., & Rosenström, T. H. (2022) · Journal of Medical Internet Research · doi:10.2196/38911 · Cited by 13
Rodrigo, H., Beukes, E. W., Andersson, G., & Manchaiah, V. (2022) · American Journal of Audiology · doi:10.1044/2022_aja-21-00270 · Cited by 9
Cote, M. P., & Lubowitz, J. H. (2024) · Arthroscopy · doi:10.1016/j.arthro.2023.12.027 · Cited by 8
Synchronous telehealth technologies in psychotherapy for depression: A meta-analysis
Osenbach (2013) · Depression and Anxiety · doi:10.1002/da.2013.30.issue-11 · Cited by 8
Rocha, A., Camacho, R., Ruwaard, J., & Riper, H. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.03.003 · Cited by 7
He, Y., Yang, L., Zhu, X., Wu, B., Zhang, S., Qian, C., & Tian, T. (2022) · doi:10.2196/preprints.40719 · Cited by 3
Danieli, M., Ciulli, T., Mousavi, M. S., Silvestri, G., Barbato, S., Di Natale, L., & Riccardi, G. (2022) · doi:10.2196/preprints.38067 · Cited by 2
Alavi, N., Yang, M., Stephenson, C., et al. (2020) · doi:10.2196/preprints.24913 · Cited by 1
Self-citationAlavi, N., Stephenson, C., & Rivera, M. (2021) · doi:10.2196/preprints.27308 · Cited by 1
Self-citation
Research works printed but not linked: 4
- 2018, Mental health in the balance: ending the healthcare disparity in Canada
- 2020, Mental Illness and Addiction: Facts and Statistics
- Covidence Systematic Review Software2022
- Lipsey, 2001, Practical meta-analysis. Practical meta-analysis, ix, 247
Cited by
Can AI replace psychotherapists? Exploring the future of mental health care
Zhang, Z., & Wang, J. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1444382 · Cited by 155
Chan, C. K. Y. (2025) · Behavioral Sciences · doi:10.3390/bs15030287 · Cited by 87
“AI-integrated tools could offer real-time responses, improve accessibility, and act as an alternative to traditional practices (Fiske et al., 2019; Gutierrez et al., 2024; Laranjo et al., 2018).”
Mentalizing Without a Mind: Psychotherapeutic Potential of Generative AI
Yirmiya, K., & Fonagy, P. (2025) · Journal of Medical Internet Research · doi:10.2196/79156 · Cited by 49
“AI technologies offer substantial promise in diagnostics, psychoeducation, mental health first aid, and professional training (Cruz-Gonzalez et al [6]; Gutierrez et al [3]; Yirmiya et al [84]; and Zhong et al [85]).”
Applications of Artificial Intelligence in Psychiatry and Psychology Education: Scoping Review
Prégent, J., Chung, V.-H.-A., Adib, I. E., Désilets, M., & Hudon, A. (2025) · JMIR Medical Education · doi:10.2196/75238 · Cited by 44
Data Privacy, Ethics, and the Role of AI in Customer Relationship Management
Kumar, A., Sharma, A., Dhanka, S., Bist, Y., Maini, S., & Bhatnagar, P. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1867-7.ch013 · Cited by 18
Metadata
- Authors
- Gutierrez, G., Stephenson, C., Eadie, J., Asadpour, K., & Alavi, N.
- Affiliations
- Queen's University (CA)
- Corresponding author
- Alavi, Nazanin
- Venue
- Frontiers in Psychiatry
- Publisher
- Frontiers Media SA
- Published
- 2024-05-07
- Volume
- 15
- Pages
- 1356773-1356773
- ISSN
- 1664-0640 (ISSN-L)
- PMC
- PMC11106393
- OpenAlex
- W4396704075
- Cited by
- 71
- References
- 109
- Citations by year
- 2026: 28 · 2025: 36 · 2024: 7
- FWCI
- 4.30
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Psychiatry (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (acceptedVersion)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- artificial intelligence, digital mental health interventions, medication adherence, treatment discontinuation, depression, anxiety, AI triage, treatment response prediction, self-guided intervention
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.