The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Characterizing Delusional Spirals through Human-LLM Chat Logs
Open accessAbstract
As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and “AI psychosis,” have emerged in global media and legal discourse. However, it remains unclear how users and chatbots interact over the course of lengthy delusional “spirals,” limiting our ability to understand and mitigate the harm. In our work, we analyze logs of conversations with LLM chatbots from 19 users who report having experienced psychological harms from chatbot use. Many of our participants come from a support group for such chatbot users. We also include chat logs from participants covered by media outlets in widely-distributed stories about chatbot-reinforced delusions. In contrast to prior work that speculates on potential AI harms to mental health, to our knowledge we present the first in-depth study of such high-profile and veridically harmful cases. We develop an inventory of 28 codes and apply it to the 391, 562 messages in the logs. Codes include whether a user demonstrates delusional thinking (15.5% of user messages), a user expresses suicidal thoughts (69 validated user messages), or a chatbot misrepresents itself as sentient (21.2% of chatbot messages). We analyze the co-occurrence of message codes. We find, for example, that messages that declare romantic interest and messages where the chatbot describes itself as sentient occur much more often in longer conversations, suggesting that these topics could promote or result from user over-engagement and that safeguards in these areas may degrade in multi-turn settings. We conclude with concrete recommendations for how policymakers, LLM chatbot developers, and users can use our inventory and conversation analysis tool to understand and mitigate harm from LLM chatbots.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Inkster, B., Sarda, S., & Subramanian, V. (2018) · doi:10.2196/preprints.12106 · Cited by 1,063
Fulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018) · doi:10.2196/preprints.9782 · Cited by 715 · cited 1×
“Therapeutic use is promoted by companies that deploy direct-to-consumer “wellness”apps, such as Wysa [45], Woebot[21], Tess [30], and Ash [83].”
Natural language processing applied to mental illness detection: a narrative review
Zhang, T., Schoene, A. M., Ji, S., & Ananiadou, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00589-7 · Cited by 476
Forecasting the onset and course of mental illness with Twitter data
Reece, A. G., Reagan, A. J., Lix, K. L. M., Dodds, P. S., Danforth, C. M., & Langer, E. J. (2017) · Scientific Reports · doi:10.1038/s41598-017-12961-9 · Cited by 401 · cited 1×
“Researchers from a variety of fields have developed computational methods to classify text for indicators of mental health symptoms, conditions, and disclosures [4, 11, 13, 56, 77, 94].”
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369
Self-citationDoes asking about suicide and related behaviours induce suicidal ideation? What is the evidence?
Dazzi, T., Gribble, R., Wessely, S., & Fear, N. T. (2014) · Psychological Medicine · doi:10.1017/s0033291714001299 · Cited by 368
Natural language processing in mental health applications using non-clinical texts
CALVO, R. A., MILNE, D. N., HUSSAIN, M. S., & CHRISTENSEN, H. (2017) · Natural Language Engineering · doi:10.1017/s1351324916000383 · Cited by 362 · cited 1×
“Researchers from a variety of fields have developed computational methods to classify text for indicators of mental health symptoms, conditions, and disclosures [4, 11, 13, 56, 77, 94].”
Althoff, T., Clark, K., & Leskovec, J. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00111 · Cited by 322
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533083 · Cited by 317 · cited 1×
“Following other work [10, 48], we designed these codes inductively, based on emergent themes after reading through the logs ourselves with an eye toward understanding both user’s delusional spirals with LLM chatbots and other mental health harms in general (§§3.2).”
InfluentialSelf-citationThe Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2021) · arXiv · doi:10.48550/arxiv.2106.15590 · Cited by 317
Self-citation
Research works printed but not linked: 13
- Substance Abuse, Mental Health Services Administration, et al. 2016. Impact of the DSM-IV to DSM-5 Changes on the National Survey on Drug Use and Health. (2016).
- American Psychiatric Association. 2022. Diagnostic and statistical manual of mental disorders DSM-5-TR (fifth edition, text revision ed.). American Psychiatric Association Publishing.
- Social Media Victims Law Center. 2025. Social Media Victims Law Center and Tech Justice Law Project lawsuits accuse ChatGPT of emotional manipulation supercharging AI delusions and acting as a “suicide coach”. https://socialmediavictims.org/press-releases/smvlc-tech-justice-law-project-lawsuits-accuse-chatgpt-of-emotional-manipulation-supercharging-ai-delusions-and-acting-as-a-suicide-coach/
- Julian De Freitas, Zeliha Oguz-Uguralp, and Ahmet Kaan-Uguralp. 2025. Emotional manipulation by AI companions. arXiv preprint arXiv:2508.19258 (2025). · arXiv
- W Bradley Knox, Katie Bradford, Samanta Varela Castro, Desmond C Ong, Sean Williams, Jacob Romanow, Carly Nations, Peter Stone, and Samuel Baker. 2025. Harmful traits of AI companions. (2025).
- Marsha Linehan. 1993. Cognitive-behavioral treatment of borderline personality disorder. Guilford press.
- Zilin Ma, Yiyang Mei, and Zhaoyuan Su. 2024. Understanding the Benefits and Challenges of Using Large Language Model-based Conversational Agents for Mental Well-being Support. 2023 (2024), 1105–1114. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10785945/
- John J McGrath, Sukanta Saha, Ali Al-Hamzawi, Jordi Alonso, Evelyn J Bromet, Ronny Bruffaerts, José Miguel Caldas-de Almeida, Wai Tat Chiu, Peter de Jonge, John Fayyad, et al. 2015. Psychotic experiences in the general population: a cross-national analysis based on 31 261 respondents from 18 countries. JAMA psychiatry 72, 7 (2015), 697–705.
- Ashish Mehta, Jared Moore, Jacy Reese Anthis, William Agnew, Eric Lin, Peggy Yin, Desmond C. Ong, Nick Haber, and Carol Dweck. 2026. The Dynamics of Delusion: Modeling Bidirectional False Belief Amplification in Human-Chatbot Dialogue. https://spirals.stanford.edu/p/dynamics Preprint.
- John E Overall and Donald R Gorham. 1988. The Brief Psychiatric Rating Scale (BPRS): recent developments in ascertainment and scaling. Psychopharmacol Bull 24, 1 (1988), 97–99.
- Joseph M. Pierre, Ben Gaeta, Govind Raghavan, and Karthik V. Sarma. 2025. “You're Not Crazy”: A Case of New-onset AI-associated Psychosis. 22, 10 (2025), 11. https://innovationscns.com/youre-not-crazy-a-case-of-new-onset-ai-associated-psychosis/
- Michael B. Robb and Supreet Mann. 2025. Talk, trust, and trade-offs: How and why teens use AI companions. https://www.commonsensemedia.org/sites/default/files/research/report/talk-trust-and-trade-offs_2025_web.pdf
- Viktor Tausk. 1919. Über die Entstehung des “Beeinflussungsapparates” in der Schizophrenie. 5 (1919), 1–33. https://pep-web.org/browse/document/izpa.005.0001a
Cited by
AI for Proactive Mental Health: A Multi-Institutional, Longitudinal Randomized Controlled Trial
Cachia, J. Y. A., Zhao, X., Hunter, J. F., Wu, D. C., Lin, E., & Freitas, J. D. (2026) · NEJM AI · doi:10.1056/aioa2501293 · Cited by 11
Self-citationAI Mental Health Support and the Case for Distributed Networked Regulation
Paterson, J. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009860796 · Cited by 1
Bariach, B., Schoenegger, P., Bhaskar, M., & Suleyman, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01294-x · Cited by 6
Digital Mental Health Research Priorities, Revisited for the AI and Large Language Model Era
Birk, M. V., Kochhar, S., Myrick, K., Schueller, S. M., & Torous, J. B. (2026) · JMIR Mental Health · doi:10.2196/104118 · Cited by 8
“AI harms can be clear, but they can also accumulate slowly over long conversations and the formation of a dangerous parasocial relationship that may be harder to detect, yet more deadly [25].”
AI ‐Associated Psychotic Phenomena: Collect Before You Classify
Augustin, M., & Torous, J. B. (2026) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.70123
Metadata
- Authors
- Moore, J., Mehta, A., Agnew, W., Anthis, J. R., Louie, R., Mai, Y., Yin, P., Cheng, M., Paech, S. J., Klyman, K., Chancellor, S., Lin, E., Haber, N., & Ong, D. C.
- Affiliations
- Stanford University (US)
- Carnegie Mellon University (US)
- University of Chicago (US)
- Independent Researcher, N/A, Australia
- Harvard University (US)
- University of Minnesota (US)
- The University of Texas at Austin (US)
- ORCID
- Venue
- Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2026-06-25
- Pages
- 7631-7674
- OpenAlex
- W7166582319
- Cited by
- 15
- References
- 45
- Citations by year
- 2026: 15
- FWCI
- 142.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- large language model chatbots, AI psychosis, suicidal ideation, chat logs, conversation analysis, multi-turn dialogue
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Characterizing Delusional Spirals through Human-LLM Chat Logs. https://app.cochat.ai/papers/10.1145/3805689.3806443
- The paper
- Moore, J., Mehta, A., Agnew, W., Anthis, J. R., Louie, R., Mai, Y., Yin, P., Cheng, M., Paech, S. J., Klyman, K., Chancellor, S., Lin, E., Haber, N., & Ong, D. C. (2026). Characterizing Delusional Spirals through Human-LLM Chat Logs. Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3805689.3806443
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.