CoChat
Sign Up
Cited by 57Digital Mental Health InterventionsArtificial Intelligence in Healthcare and EducationSocial Robot Interaction and HRI

Artificial intelligence conversational agents in mental health: Patients see potential, but prefer humans in the loop

Open access
Lee, Hyein S., Wright, Colton, Ferranto, Julia, Buttimer, Jessica, Palmer, Clare E., Welchman, Andrew, Mazor, Kathleen M., Fisher, Kimberly A. +4 more·2025·Frontiers in Psychiatry
DOI 10.3389/fpsyt.2024.1505024PMC PMC11826059
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Digital mental health interventions, such as artificial intelligence (AI) conversational agents, hold promise for improving access to care by innovating therapy and supporting delivery. However, little research exists on patient perspectives regarding AI conversational agents, which is crucial for their successful implementation. This study aimed to fill the gap by exploring patients’ perceptions and acceptability of AI conversational agents in mental healthcare. Methods: Adults with self-reported mild to moderate anxiety were recruited from the UMass Memorial Health system. Participants engaged in semi-structured interviews to discuss their experiences, perceptions, and acceptability of AI conversational agents in mental healthcare. Anxiety levels were assessed using the Generalized Anxiety Disorder scale. Data were collected from December 2022 to February 2023, and three researchers conducted rapid qualitative analysis to identify and synthesize themes. Results: The sample included 29 adults (ages 19-66), predominantly under age 35, non-Hispanic, White, and female. Participants reported a range of positive and negative experiences with AI conversational agents. Most held positive attitudes towards AI conversational agents, appreciating their utility and potential to increase access to care, yet some also expressed cautious optimism. About half endorsed negative opinions, citing AI’s lack of empathy, technical limitations in addressing complex mental health situations, and data privacy concerns. Most participants desired some human involvement in AI-driven therapy and expressed concern about the risk of AI conversational agents being seen as replacements for therapy. A subgroup preferred AI conversational agents for administrative tasks rather than care provision. Conclusions: AI conversational agents were perceived as useful and beneficial for increasing access to care, but concerns about AI’s empathy, capabilities, safety, and human involvement in mental healthcare were prevalent. Future implementation and integration of AI conversational agents should consider patient perspectives to enhance their acceptability and effectiveness.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fpsyt.2024.1505024

Cites

60 of 61 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Interview-based experiences of users with mental health issues using LLM-CBs for mental health show that beyond positive attitudes, a wider range of users raise concerns related to data privacy, inappropriate advice, hallucinations, lack of genuine empathy, and the need for a human-in-the-loop.

    “Interview-based experiences of users with mental health issues using LLM-CBs for mental health show that beyond positive attitudes, a wider range of users raise concerns related to data privacy, inappropriate advice, hallucinations, lack of genuine empathy, and the need for a human-in-the-loop78–80.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [78]

    The Results and Discussion (pages unavailable in our copy) report positive but hesitant views alongside concerns about data privacy, lack of empathy, technical reliability, and the need for human involvement. They do not report participants raising hallucinations or inappropriate advice as distinct concerns. The Methods > Procedure also says the prototype used tree-based dialogue and produced no generative responses, so these interviews cannot specifically establish users’ experiences with LLM-based mental-health chatbots. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

3 of the 64 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. [57]Zhang Z Genc Y Xing A Wang D Fan X Citardi D. Lay individuals’ perceptions of artificial intelligence (AI)-empowered healthcare systems. Proc Assoc Inf Sci Technol. (2020) 57: e326. doi: 10.1002/pra2.v57.1 · DOI
  2. [58]Draxler F Buschek D Tavast M Hämäläinen P Schmidt A Kulshrestha J. Gender, Age, and Technology Education Influence the Adoption and Appropriation of LLMs (2023). Available online at: https://arxiv.org/abs/2310.06556 (Accessed January 29, 2024). · arXiv

Cited by

Sorted by citations

Related

Authors
Lee, H. S., Wright, C., Ferranto, J., Buttimer, J., Palmer, C. E., Welchman, A., Mazor, K. M., Fisher, K. A., Smelson, D., O’Connor, L., Fahey, N., & Soni, A.
Affiliations
  • University of Massachusetts Chan Medical School (US)
  • ieso Digital Health (UK) Ltd. (United Kingdom) (GB)
Corresponding author
Soni, Apurv
ORCID
Venue
Frontiers in Psychiatry
Publisher
Frontiers Media SA
Published
2025-01-31
Volume
15
Pages
1505024-1505024
ISSN
1664-0640 (ISSN-L)
DOI
10.3389/fpsyt.2024.1505024
PMC
PMC11826059
OpenAlex
W4407041715
Cited by
55
References
64
Citations by year
2026: 43 · 2025: 12
FWCI
44.01
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Psychiatry (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (acceptedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
mental health, patient perspectives, acceptability, anxiety, digital mental health interventions, human-AI collaboration, empathy, data privacy, health care access

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review