Evaluating Generative AI in Mental Health: Systematic Review of Capabilities and Limitations
Open accessAbstract
Background: The global shortage of mental health professionals, exacerbated by increasing mental health needs post COVID-19, has stimulated growing interest in leveraging large language models to address these challenges. Objectives: This systematic review aims to evaluate the current capabilities of generative artificial intelligence (GenAI) models in the context of mental health applications. Methods: A comprehensive search across 5 databases yielded 1046 references, of which 8 studies met the inclusion criteria. The included studies were original research with experimental designs (eg, Turing tests, sociocognitive tasks, trials, or qualitative methods); a focus on GenAI models; and explicit measurement of sociocognitive abilities (eg, empathy and emotional awareness), mental health outcomes, and user experience (eg, perceived trust and empathy). Results: The studies, published between 2023 and 2024, primarily evaluated models such as ChatGPT-3.5 and 4.0, Bard, and Claude in tasks such as psychoeducation, diagnosis, emotional awareness, and clinical interventions. Most studies used zero-shot prompting and human evaluators to assess the AI responses, using standardized rating scales or qualitative analysis. However, these methods were often insufficient to fully capture the complexity of GenAI capabilities. The reliance on single-shot prompting techniques, limited comparisons, and task-based assessments isolated from a context may oversimplify GenAI’s abilities and overlook the nuances of human–artificial intelligence interaction, especially in clinical applications that require contextual reasoning and cultural sensitivity. The findings suggest that while GenAI models demonstrate strengths in psychoeducation and emotional awareness, their diagnostic accuracy, cultural competence, and ability to engage users emotionally remain limited. Users frequently reported concerns about trustworthiness, accuracy, and the lack of emotional engagement. Conclusions: Future research could use more sophisticated evaluation methods, such as few-shot and chain-of-thought prompting to fully uncover GenAI’s potential. Longitudinal studies and broader comparisons with human benchmarks are needed to explore the effects of GenAI-integrated mental health care.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing scholarly literature on GenAI's application in healthcare often focuses almost exclusively on technical performance indicators such as accuracy rates, time efficiency, and comparative error reduction against human benchmarks.
“Existing scholarly literature17–22 on GenAI’s application in healthcare often focuses almost exclusively on technical performance indicators such as accuracy rates, time efficiency, and comparative error reduction against human benchmarks.”
From Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications – A Systematic Review (2026) · cites this paper as [22]
The Results section (Evaluation of GenAI Capabilities; page unavailable) reports task-based evaluations that included accuracy, supporting part of the claim. But it also lists empathy, engagement, and ethical considerations among the metrics, says most studies had no comparison group, and does not report a focus on time efficiency or error reduction against humans. The review covers mental health applications, not healthcare literature generally. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · cited 1× · see citing works
“The database results were imported into Rayyan software [18].”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“Current models are largely trained on data from Western, English-speaking sources, which introduce inherent biases into their responses [40,41].”
Santomauro, D. F., Mantilla Herrera, A. M., Shadid, J., Zheng, P., Ashbaugh, C., Pigott, D. M., Abbafati, C., Adolph, C., Amlag, J. O., Aravkin, A. Y., et al. (2021) · The Lancet · doi:10.1016/s0140-6736(21)02143-7 · Cited by 5,444 · see citing works
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737 · cited 1× · see citing works
“We used the Mixed Methods Appraisal Tool (MMAT; version 2018)to evaluate the methodological quality of studies included in this review [19].”
Transtheoretical therapy: Toward a more integrative model of change.
Prochaska, J. O., & DiClemente, C. C. (1982) · Psychotherapy: Theory, Research & Practice · doi:10.1037/h0088437 · Cited by 3,922 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 2× · see citing works
“More sophisticated prompting methods, such as few-shot, CoT, and a combination of the 2, have been shown to significantly improve model performance in multiple reasoning tasks [34-36].”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752 · see citing works
Stade, E. C., Stirman, S. W., Ungar, L. H., Boland, C. L., Schwartz, H. A., Yaden, D. B., Sedoc, J., DeRubeis, R. J., Willer, R., & Eichstaedt, J. C. (2024) · npj Mental Health Research · doi:10.1038/s44184-024-00056-z · Cited by 391 · cited 1× · see citing works
“…burden and increase clinician efficiency through LLM-assisted clinical notes writing, formulating differential diagnoses, drafting personalized treatment plans, drawing insights from patient chart data, providing on-demand coaching and companion-ship, and, ultimately, providing therapy [10,11].”
ChatGPT outperforms humans in emotional awareness evaluations
Elyoseph, Z., Hadar-Shoval, D., Asraf, K., & Lvovsky, M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1199058 · Cited by 324 · see citing works
Automatic Chain of Thought Prompting in Large Language Models
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274 · see citing works
Timmons, A. C., Duong, J. B., Simo Fiallo, N., Lee, T., Vo, H. P. Q., Ahle, M. W., Comer, J. S., Brewer, L. C., Frazier, S. L., & Chaspari, T. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916221134490 · Cited by 263 · see citing works
Loneliness and suicide mitigation for students using GPT3-enabled chatbots
Maples, B., Cerit, M., Vishwanath, A., & Pea, R. (2024) · npj Mental Health Research · doi:10.1038/s44184-023-00047-6 · Cited by 250 · cited 1× · see citing works
“In addition, some individuals have begun using ChatGPT as a therapeutic companion or informal therapist [13].”
Khawaja, Z., & Bélisle-Pipon, J.-C. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1278186 · Cited by 214 · see citing works
Assessing the Effectiveness of ChatGPT in Delivering Mental Health Support: A Qualitative Study
Alanezi, F. (2024) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s447368 · Cited by 139 · see citing works
Should ChatGPT Be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · doi:10.2139/ssrn.4627814 · Cited by 133 · see citing works
King, D. R., Nanda, G., Stoddard, J., Dempsey, A., Hergert, S., Shore, J. H., & Torous, J. (2023) · Current Psychiatry Reports · doi:10.1007/s11920-023-01477-x · Cited by 63 · see citing works
Flagged first, then by citations
Research works printed but not linked: 27
- [2]Malerbi FK Nakayama LF Gayle Dychiao R et al Digital education for the deployment of artificial intelligence in health care J Med Internet Res Jun 22 2023 25 e43333 doi 10.2196/43333 Medline 37347537 PMC10337407 · DOI
- [3]Zahoor S Lali IU Khan MA Javed K Mehmood W Breast cancer detection and classification using traditional computer vision techniques: a comprehensive review CMIR 01 12 2021 16 10 1187 1200 doi 10.2174/1573405616666200406110547 32250226 · DOI
- [4]Ji S Pan S Li X Cambria E Long G Huang Z Suicidal ideation detection: a review of machine learning methods and applications IEEE Trans Comput Soc Syst 2021 8 1 214 226 doi 10.1109/TCSS.2020.3021467 · DOI
- [5]De Angelis L Baglivo F Arzilli G et al ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health Front Public Health 2023 11 1166120 doi 10.3389/fpubh.2023.1166120 Medline 37181697 PMC10166793 · DOI · Paper page
- [6]Semrl N Feigl S Taumberger N et al AI language models in human reproduction research: exploring ChatGPT’s potential to assist academic writing Hum Reprod Dec 4 2023 38 12 2281 2288 doi 10.1093/humrep/dead207 Medline 37833847 · DOI
- [7]Lee TK Artificial intelligence and posthumanist translation: ChatGPT versus the translator Appl Linguist Rev Nov 26 2024 15 6 2351 2372 doi 10.1515/applirev-2023-0122 · DOI
- [10]Singh OP Artificial intelligence in the era of ChatGPT—opportunities and challenges in mental health care Indian J Psychiatry Mar 2023 65 3 297 298 doi 10.4103/indianjpsychiatry.indianjpsychiatry_112_23 Medline 37204980 PMC10187878 · DOI · Paper page
- [15]Obradovich N Khalsa SS Khan W et al Opportunities and risks of large language models in psychiatry NPP Digit Psychiatry Neurosci 2024 2 1 1 8 doi 10.1038/s44277-024-00010-z Medline 39554888 PMC11566298 · DOI
- [16]Falender C Shafranske EP Clinical Supervision: A Competency-Based Approach American Psychological Association 2004 doi 10.1037/0000243-000 · DOI
- [17]Moher D Liberati A Tetzlaff J Altman DG PRISMA Group Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement Ann Intern Med Aug 18 2009 151 4 264 269 doi 10.7326/0003-4819-151-4-200908180-00135 Medline 19622511 · DOI · Paper page
- [20]Razdan S Siegal AR Brewer Y Sljivich M Valenzuela RJ Assessing ChatGPT’s ability to answer questions pertaining to erectile dysfunction: can our patients trust it? Int J Impot Res Nov 2024 36 7 734 740 doi 10.1038/s41443-023-00797-z Medline 37985815 · DOI
- [22]Maurya RK Montesinos S Bogomaz M DeDiego AC Assessing the use of ChatGPT as a psychoeducational tool for mental health practice Couns Psychother Res Mar 2025 25 1 doi 10.1002/capr.12759 · DOI
- [24]Elyoseph Z Levkovich I Comparing the perspectives of generative AI, mental health experts, and the general public on schizophrenia recovery: case vignette study JMIR Ment Health Mar 18 2024 11 e53043 doi 10.2196/53043 Medline 38533615 PMC11004608 · DOI
- [25]Balamurali BT Chen JM Performance assessment of ChatGPT versus Bard in detecting Alzheimer’s dementia Diagnostics (Basel) Apr 15 2024 14 8 817 doi 10.3390/diagnostics14080817 Medline 38667463 PMC11048951 · DOI
- [26]Hadar-Shoval D Elyoseph Z Lvovsky M The plasticity of ChatGPT’s mentalizing abilities: personalization for personality structures Front Psychiatry 2023 14 1234397 doi 10.3389/fpsyt.2023.1234397 Medline 37720897 PMC10503434 · DOI
- [27]Kojima T Gu SS Reid M Matsuo Y Iwasawa Y Large language models are zero-shot reasoners arXiv Preprint posted online on 05 24 2022 doi 10.48550/arXiv.2205.11916 · DOI · Paper page
- [28]Gore S Dove E Ethical considerations in the use of artificial intelligence in counselling and psychotherapy training: a student stakeholder perspective—a pilot study Couns Psychother Res Mar 2025 25 1 2024 doi 10.1002/capr.12770 · DOI
- [29]Utay J Megan M Guided imagery as an effective therapeutic technique: a brief review of its history and efficacy research J Instr Psychol 2006 33 1 40 43 URL https://www.proquest.com/openview/26bdce6b33c205ccdccdc5969b38bfbc/1?cbl=48173&pq-origsite=gscholar Accessed 10-09-2024
- [30]Weigensberg MJ Lane CJ Winners O et al Acute effects of stress-reduction Interactive Guided Imagery(SM) on salivary cortisol in overweight Latino adolescents J Altern Complement Med Mar 2009 15 3 297 303 doi 10.1089/acm.2008.0156 Medline 19250005 PMC2838612 · DOI
- [31]Wei Q Yao Z Cui Y Wei B Jin Z Xu X Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis J Biomed Inform Mar 2024 151 104620 doi 10.1016/j.jbi.2024.104620 Medline 38462064 · DOI
- [32]Johnson D Goodman R Patrinely J et al Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model Res Sq Feb 28 2023 28 rs.3.rs-2566942 doi 10.21203/rs.3.rs-2566942/v1 Medline 36909565 PMC10002821 · DOI · Paper page
- [33]Yeo YH Samaan JS Ng WH et al Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma Clin Mol Hepatol 07 2023 29 3 721 732 doi 10.3350/cmh.2023.0089 Medline 36946005 PMC10366809 · DOI
- [34]Wei J Wang X Schuurmans D et al Chain-of-thought prompting elicits reasoning in large language models arXiv Preprint posted online on 01 28 2022 doi 10.48550/arXiv.2201.11903 · DOI · Paper page
- [37]Ahsan MM Luna SA Siddique Z Machine-learning-based disease diagnosis: a comprehensive review Healthcare (Basel) Mar 15 2022 10 3 541 doi 10.3390/healthcare10030541 Medline 35327018 PMC8950225 · DOI · Paper page
- [39]Zhang Y Yuan Y Yao ACC Meta prompting for AI systems arXiv Preprint posted online on Nov 22 2023 doi 10.48550/arXiv.2311.11482 · DOI
- [43]Ryder AG Yang J Zhu X et al The cultural shaping of depression: somatic symptoms in China, psychological symptoms in North America? J Abnorm Psychol 05 2008 117 2 300 313 doi 10.1037/0021-843X.117.2.300 Medline 18489206 · DOI
- [44]Sue S Cultural competency: from philosophy to research and practice J Community Psychol Mar 2006 34 2 237 245 doi 10.1002/jcop.20095 · DOI · Paper page
Cited by
Sorted by citations
Al-Anezi, F. M. (2026) · Journal of Healthcare Leadership · doi:10.2147/jhl.s590498 · Cited by 6 · see citing works
- Authors
- Wang, L., Bhanushali, T., Huang, Z., Yang, J., Badami, S., & Hightow-Weidman, L.
- Affiliations
- Florida State University (US)
- University of Washington (US)
- Northeastern University (US)
- Columbia University (US)
- Corresponding author
- Wang, Liying
- Venue
- JMIR Mental Health
- Publisher
- JMIR Publications Inc.
- Published
- 2025-05-15
- Volume
- 12
- Pages
- e70014-e70014
- ISSN
- 2368-7959 (ISSN-L)
- PMC
- PMC12097452
- OpenAlex
- W4407677752
- Cited by
- 73
- References
- 16
- Citations by year
- 2026: 63 · 2025: 10
- FWCI
- 4.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Mental Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- generative AI, large language models, mental health, AI evaluation, psychoeducation, emotional awareness, AI-assisted diagnosis, cultural competence, user trust, human-AI interaction, ChatGPT, Claude, bards, mental health workforce shortage
- MeSH terms
- COVID-19, Artificial Intelligence, Mental Disorders, Humans, Mental Health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
