CoChat
Sign Up
Cited by 88Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIDigital Mental Health Interventions

Will Generative Artificial Intelligence Chatbots Generate Delusions in Individuals Prone to Psychosis?

Open access
Østergaard, Søren Dinesen·2023·Schizophrenia Bulletin
DOI 10.1093/schbul/sbad128PMC PMC10686326
Open this paper in CoChat — free accountOpen PDF

Abstract

Generative artificial intelligence (AI) chatbots such as ChatGPT, GPT-4, Bing, and Bard are currently receiving substantial attention.1,2 Indeed, in January 2023, just two months after its launch, ChatGPT reached 100 million monthly active users, which set the record for the fastest-growing user base for an internet application. For comparison, it took Tiktok 9 months and Instagram 2 and a half years to reach a similar number of active users.2 As one of the many users, I have mainly been “testing” ChatGPT from a psychiatric perspective and I see both possibilities and challenges in this regard. In terms of possibilities, it is my impression that ChatGPT generally provides fairly accurate and balanced answers when asked about mental illness. For instance, when pretending to be an individual suffering from depression, describing my symptoms to ChatGPT, it answered that they are compatible with depression and suggested that I should seek professional help. Similarly, when asking about various treatments for mental disorders, ChatGPT generally provided useful answers. This was also true when I asked about electroconvulsive therapy (ECT), which was a positive surprise given the amount of misinformation on ECT on the internet3—and the internet being a central part of the corpus on which ChatGPT was trained.4 There are of course important potential pitfalls in this context. For instance, depending on the questions asked, generative AI chatbots may provide information that is wrong or maybe misunderstood by a person with mental illness in need of medical attention, who does then not seek appropriate help. However, from my strictly informal and non-exhaustive test, I am cautiously optimistic that generative AI chatbots may be able to support future psychoeducation initiatives in a world where the demand for such initiatives is hard to meet using more conventional methods. Time will tell how this turns out. In terms of challenges posed by generative AI in the field of psychiatry, there are many. Most importantly perhaps, there are rising concerns that malicious actors may use generative AI to create misinformation at a scale that will be very difficult to counter.5 While this concern is, by no means, specific to psychiatry, but rather represents a general challenge for societies more broadly, it can be argued that individuals with mental illness may be particularly sensitive to such misinformation. There is, however, also a potential challenge that is specific to psychiatry. Indeed, there are prior accounts of people becoming delusional (de novo) when engaging in chat conversations with other people on the internet.6 While establishing causality in such cases is of course inherently difficult, it seems plausible for this to happen for individuals prone to psychosis. I would argue that the risk of something similar occurring due to interaction with generative AI chatbots is even higher. Specifically, the correspondence with generative AI chatbots such as ChatGPT is so realistic that one easily gets the impression that there is a real person at the other end—while, at the same time, knowing that this is, in fact, not the case. In my opinion, it seems likely that this cognitive dissonance may fuel delusions in those with increased propensity towards psychosis. Furthermore, even when having accepted that you are corresponding with a computer program, the mystery does not stop: How (on earth) can a computer program respond so well to all sorts of questions? If doing a bit of reading on this topic, you will come to realize that the answer to this question is that nobody really knows for sure—as there is a substantial “black box” element to it.5 In other words, the inner workings of generative AI also leave ample room for speculation/paranoia. Finally, there are reports of people having had rather confrontational encounters with generative AI chatbots, who “fell in love” or indirectly opposed/threatened them.7 On this background, I provide 5 examples of potential delusions (from the perspective of the individuals experiencing them) that could plausibly arise due to interaction with generative AI chatbots: Delusion of persecution: “This chatbot is not controlled by a tech company, but by a foreign intelligence agency using it to spy on me. I have formatted the hard disk on my computer as a consequence, but my roommate keeps using the chatbot, so the spying continues.” Delusion of reference: “It is evident from the words used in this series of answers that the chatbot is writing to me personally and specifically with a message, the content of which I am unfortunately not allowed to convey to you.” Thought broadcasting: “Many of the chatbot’s answers to its users are in fact my thoughts being transmitted via the internet.” Delusion of guilt: “Due to my many questions to the chatbot, I have taken up time from people who really needed the chatbot’s help, but could not access it. I also think that I have somehow harmed the chatbo

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/schbul/sbad128

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For such users, the alternative belief that the chatbot is a non-conscious machine may strike the agent as alien or incompatible with their background assumptions about reliable indicators of consciousness, rendering it explanatorily unavailable to them.

    “the alternative belief, that the chatbot is a non-conscious machine, may strike the agent as alien or incompatible with their background assumptions about reliable indicators of consciousness (e.g., sophisticated verbal responding) (Østergaard, 2023), rendering it explanatorily unavailable to them.”

    From Are Attributions of Consciousness to AI Chatbots Epistemically Innocent? (2026) · cites this paper as Østergaard, 2023

    In the main text (page not available in our copy), Østergaard says a chatbot’s realistic responses can make it seem as though a real person is responding, despite the user knowing otherwise, and that this dissonance may fuel delusions in people prone to psychosis. The paper does not discuss users’ assumptions about indicators of consciousness or say that the machine explanation becomes unavailable to them; it also considers users who accept that they are corresponding with a computer program. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • LLM-CBs may themselves be integrated into delusional belief systems, which could be validated directly or indirectly by the LLM-CBs.

    “LLM-CBs may themselves be integrated into delusional belief systems, which could be validated directly or indirectly by the LLM-CBs (e.g., believing the chatbot is targeting the user with important messages in the context of delusions of reference)19.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [19]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Østergaard, S. D.
Affiliations
Aarhus University (DK)
Corresponding author
Østergaard, Søren Dinesen
ORCID
Østergaard, Søren Dinesen
Venue
Schizophrenia Bulletin
Publisher
Oxford University Press (OUP)
Published
2023-08-25
Volume
49
Issue
6
Pages
1418-1419
ISSN
  • 0586-7614 (ISSN-L)
  • 1745-1701
DOI
10.1093/schbul/sbad128
PMC
PMC10686326
OpenAlex
W4386158236
Cited by
87
References
3
Citations by year
2026: 66 · 2025: 18 · 2024: 3
FWCI
3.47
Type
journal article
Language
en
Open access
Licence
academic.oup.com/pages/standard-publication-reuse-rights (vor)
Locations
  • Schizophrenia Bulletin (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • Novo Nordisk Foundation (NNF20SA0062874)
  • Lundbeck Foundation (R358-2020-2341, R344-2020-1073)
  • Danish Cancer Society (R283-A16461)
  • Strengthening of Health Science (1-36-72-4-20)
  • Danish Agency for Digitization Investment Fund for New Technologies (2020-6720)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
generative AI, AI chatbots
MeSH terms
Artificial Intelligence, Delusions, Humans, Psychotic Disorders

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review