Cited by 214AI in Service InteractionsDigital Mental Health InterventionsSocial Robot Interaction and HRI

Chatbots and mental health: Insights into the safety of generative AI

Open access
De Freitas, Julian, Uğuralp, Ahmet Kaan, Oğuz‐Uğuralp, Zeliha, Puntoni, Stefano·2024·Journal of Consumer Psychology

Abstract

Chatbots are now able to engage in sophisticated conversations with consumers. Due to the “black box” nature of the algorithms, it is impossible to predict in advance how these conversations will unfold. Behavioral research provides little insight into potential safety issues emerging from the current rapid deployment of this technology at scale. We begin to address this urgent question by focusing on the context of mental health and “companion AI”: Applications designed to provide consumers with synthetic interaction partners. Studies 1a and 1b present field evidence: Actual consumer interactions with two different companion AIs. Study 2 reports an extensive performance test of several commercially available companion AIs. Study 3 is an experiment testing consumer reaction to risky and unhelpful chatbot responses. The findings show that (1) mental health crises are apparent in a nonnegligible minority of conversations with users; (2) companion AIs are often unable to recognize, and respond appropriately to, signs of distress; and (3) consumers display negative reactions to unhelpful and risky chatbot responses, highlighting emerging reputational risks for generative AI companies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/jcpy.1393

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A three-study empirical analysis found that LLM-CBs often show difficulties recognizing users' mental distress.

    “A three-study empirical analysis found that LLM-CBs often show difficulties recognizing users’ mental distress, which may lead to inappropriate responses75.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [75]

    The Conclusions (page not available) say that Study 2 found companion AIs often unable to recognize signs of distress and mental health issues, and they describe the paper’s three numbered studies. The copy describes the tested systems as companion AIs powered by black-box algorithms, but does not establish that they are specifically LLM-based chatbots. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • More people are using wellness apps to discuss mental health crises.

    “More people are using wellness apps to discuss mental health crises [40], despite the fact that these apps are often not intended to be used for such purposes.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Bendig, E., Erb, B., Schulze-Thuesing, L., & Baumeister, H. (2019). The next generation: Chatbots in clinical psychology and psychotherapy to Foster mental health-a scoping review. Verhaltenstherapie, 29, 1-13.

Cited by

Sorted by citations

Authors
De Freitas, J., Uğuralp, A. K., Oğuz‐Uğuralp, Z., & Puntoni, S.
Affiliations
  • Marketing Harvard Business School Boston Massachusetts USA
  • Bilkent University (TR)
  • Psychology Bilkent University Ankara Turkey
  • University of Pennsylvania (US)
Venue
Journal of Consumer Psychology
Publisher
Wiley
Published
2023-12-19
Volume
34
Issue
3
Pages
481-491
ISSN
  • 1057-7408 (ISSN-L)
  • 1532-7663
OpenAlex
W4388194725
Cited by
211
References
64
Citations by year
2026: 105 · 2025: 85 · 2024: 20 · 2023: 1
FWCI
19.89
Type
journal article
Language
en
Open access
Locations
  • Journal of Consumer Psychology (publishedVersion)
  • SSRN Electronic Journal (acceptedVersion)
  • Bilkent University Institutional Repository (Bilkent University) (submittedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
chatbots, AI companions, mental health, generative AI safety, reputational risk

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review