CoChat
Sign Up
Cited by 45Ethics and Social Impacts of AIDigital Mental Health InterventionsInnovative Human-Technology Interaction

From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agents

Open access
Chandra, Mohit, Naik, Suchismita, Ford, Denae, Okoli, Ebele, De Choudhury, Munmun, Ershadi, Mahsa, Ramos, Gonzalo, Hernandez, Javier +3 more·2025·Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3715275.3732063
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent gains in popularity of AI conversational agents have led to their increased use for improving productivity and supporting well-being.While previous research has aimed to understand the risks associated with interactions with AI conversational agents, these studies often fall short in capturing the lived experiences of individuals.Additionally, psychological risks have often been presented as a sub-category within broader AI-related risks in past taxonomy works, leading to under-representation of the impact of psychological risks of AI use.To address these challenges, our work presents a novel risk taxonomy focusing on psychological risks of using AI gathered through the lived experiences of individuals.We employed a mixed-method approach, involving a comprehensive survey with 283 people with lived mental health experience and workshops involving experts with lived experience to develop a psychological risk taxonomy.Our taxonomy features 19 AI behaviors, 21 negative psychological impacts, and 15 contexts related to individuals.Additionally, we propose a novel multi-path vignettebased framework for understanding the complex interplay between * Work done during internship at Microsoft Research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3715275.3732063

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Research grounded in lived experiences of this extreme user group remains notably absent.

    “Research grounded in lived experiences of this extreme user group [5, 8] remains notably absent.”

    From 'AI-Induced Delusional Spirals': Understanding Lived Experiences During Maladaptive Human-Chatbot Interactions (2026) · cites this paper as [5]

    The Introduction (page not provided) identifies people with lived mental-health experience as an “extreme user group” and says this study is grounded in their experiences, using a survey of 283 people and workshops with seven experts. Although the paper identifies a gap in earlier risk taxonomies, it presents itself as research filling that gap; it does not support the claim that such research remains absent. (LLM, conf=0.87)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Individuals with mental health problems using LLM-CB companions reported various negative experiences, including harmful/inappropriate content, manipulation or control attempts, trust or safety violations, self-perceived overreliance and emotional attachment, reinforcement of false beliefs, social withdrawal, distress, regret, intrusion, discrimination, increases of mental health issues, and loss.

    “Chandra et al. surveyed 283 individuals with mental health problems on LLM-CB companion use and found various negative experiences (e.g., harmful/inappropriate content, manipulation or control attempts, trust or safety violations, self-perceived overreliance and emotional attachment, reinforcement of false beliefs, social withdrawal, distress, regret, intrusion, discrimination, increases of mental”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [125]

    Sections 3.1.2 and 3.1.4 describe a survey of 283 people who self-identified as having lived experience in mental health; 279 participants’ scenarios were analyzed. Sections 3.2.1 and 3.2.2 report the cited types of negative experiences. The study examined AI conversational-agent use broadly, however—not specifically companion use—and its most frequent reported purposes were advice, research, and learning. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Within HCI, the lived experience approach has been continuously applied to recent work on human-centered AI design and on identifying psychological risks in conversational AI interactions.

    “Within the human-computer interaction (HCI) community, this approach has been continuously applied to recent work on human-centered AI design [14] and on identifying psychological risks in conversational AI interactions [5].”

    From 'AI-Induced Delusional Spirals': Understanding Lived Experiences During Maladaptive Human-Chatbot Interactions (2026) · cites this paper as [5]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
  2. Ahmet Adalıer and Emre Balkan. 2012. The relationship between internet addiction and psychological symptoms. International Journal of Global Education (IJGE) ISSN: 2146-9296 1, 2 (2012).
  3. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21). 2633-2650.
  4. Janet X Chen, Allison McDonald, Yixin Zou, Emily Tseng, Kevin A Roundy, Acar Tamersoy, Florian Schaub, Thomas Ristenpart, and Nicola Dell. 2022. Trauma- informed computing: Towards safer technology experiences for all. In Proceed- ings of the 2022 CHI conference on human factors in computing systems. 1-20.
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Se- bastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1-113.
  6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answer- ing? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018). · arXiv
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021). · arXiv
  8. Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Ver- ena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, et al. 2024. The ethics of advanced ai assistants. arXiv preprint arXiv:2404.16244 (2024). · arXiv
  9. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356 (2022). · arXiv
  10. Million By 2030 -grandviewresearch.com. https://www.grandviewresearch.com/press- release/global-chatbot-market. [Accessed 30-11-2024].
  11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language un- derstanding. arXiv preprint arXiv:2009.03300 (2020). · arXiv
  12. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2024. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems 36 (2024).
  13. Liu Yi Lin, Jaime E Sidani, Ariel Shensa, Ana Radovic, Elizabeth Miller, Jason B Colditz, Beth L Hoffman, Leila M Giles, and Brian A Primack. 2016. Association between social media use and depression among US young adults. Depression and anxiety 33, 4 (2016), 323-331.
  14. Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. 2024. Stable bias: Evaluating societal representations in diffusion models. Advances in Neural Information Processing Systems 36 (2024).
  15. Sharan B Merriam et al. 2002. Introduction to qualitative research. Qualitative research in practice: Examples for discussion and analysis 1, 1 (2002), 1-17.
  16. Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. 2023. Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses. arXiv preprint arXiv:2305.11662 (2023). · arXiv
  17. Pooja V Pathak and Vidhi D Mehta. 2024. Intelligent Conversations: A Theoreti- cal Framework for Understanding Natural Language Processing within Artificial Intelligence Systems. International Journal For Multidisciplinary Research (2024). https://api.semanticscholar.org/CorpusID:271252017
  18. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoff- mann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saf- fron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cy- prien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling Language Models: Meth- ods, Analysis & Insights from Training Gopher. arXiv:2112.11446 [cs.CL] https://arxiv.org/abs/2112.11446 · arXiv
  19. Leonardo Ranaldi and Giulia Pucci. 2023. When Large Language Models contra- dict humans? Large Language Models' Sycophantic Behaviour. arXiv preprint arXiv:2311.09410 (2023). · arXiv
  20. Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. 2023. Explaining legal concepts with augmented large language models (gpt-4). arXiv preprint arXiv:2306.09525 (2023). · arXiv
  21. Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. 2022. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv preprint arXiv:2210.01790 (2022). · arXiv
  22. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057 (2022). · arXiv
  23. Tamara Sims, Andrew E Reed, and Dawn C Carr. 2017. Information and com- munication technology use is related to higher well-being among the oldest-old. Journals of Gerontology Series B: Psychological Sciences and Social Sciences 72, 5 (2017), 761-770.
  24. Jaswinder Singh. 2022. Deepfakes: The Threat to Data Authenticity and Public Trust in the Age of AI-Driven Manipulation of Visual and Audio Content. Journal of AI-Assisted Scientific Discovery 2, 1 (2022), 428-467.
  25. Inhwa Song, Sachin R Pendse, Neha Kumar, and Munmun De Choudhury. 2024. The typing cure: Experiences with large language model chatbots for mental health support. arXiv preprint arXiv:2401.14362 (2024). · arXiv
  26. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070 (2024). · arXiv
  27. Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev. 2023. Beyond memorization: Violating privacy via inference with large language models. arXiv preprint arXiv:2310.07298 (2023). · arXiv
  28. Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2019. LAMOL: LAnguage MOdeling for Lifelong Language Learning. arXiv:1909.03329 [cs.CL] https: //arxiv.org/abs/1909.03329 · arXiv
  29. Yan Tao, Olga Viberg, Ryan S Baker, and Rene F Kizilcec. 2023. Auditing and mitigating cultural bias in llms. arXiv preprint arXiv:2311.14096 (2023). · arXiv
  30. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2024. Language models don't always say what they think: unfaithful explanations in chain-of- thought prompting. Advances in Neural Information Processing Systems 36 (2024).
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models. Advances in neural information processing systems 35 (2022), 24824-24837.
  32. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https://arxiv. org/pdf/2305.10601. pdf (2023).
  33. Yunpu Zhao, Rui Zhang, Junbin Xiao, Changxin Ke, Ruibo Hou, Yifan Hao, Qi Guo, and Yunji Chen. 2024. Towards Analyzing and Mitigating Sycophancy in Large Vision-Language Models. arXiv preprint arXiv:2408.11261 (2024). · arXiv
  34. Jiawei Zhou, Amy Z Chen, Darshi Shah, Laura Schwab Reese, and Munmun De Choudhury. 2024. " It's a conversation, not a quiz": A Risk Taxonomy and Re- flection Tool for LLM Adoption in Public Health. arXiv preprint arXiv:2411.02594 (2024). · arXiv

Cited by

Sorted by citations

Related

Authors
Chandra, M., Naik, S., Ford, D., Okoli, E., De Choudhury, M., Ershadi, M., Ramos, G., Hernandez, J., Bhattacharjee, A., Warreth, S., & Suh, J.
Affiliations
  • Georgia Institute of Technology (US)
  • Purdue University West Lafayette (US)
  • Microsoft (United States) (US)
  • School of Interactive Computing, Georgia Institute of Technology, Atlanta, Georgia, USA
  • Microsoft (Canada) (CA)
  • University of Toronto (CA)
  • Microsoft (Ireland) (IE)
ORCID
Venue
Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2025-06-23
Pages
975-1004
DOI
10.1145/3715275.3732063
OpenAlex
W4411550426
Cited by
44
References
67
Citations by year
2026: 34 · 2025: 10
FWCI
44.69
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Microsoft Research
  • Microsoft
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
conversational AI, psychological risk, risk taxonomy, lived experience, mental health, AI behavior, expert workshop, well-being

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review