CoChat
Sign Up
PreprintCited by 2Digital Mental Health InterventionsSocial Robot Interaction and HRIInnovative Human-Technology Interaction

AI Chatbots for Mental Health: Values and Harms from Lived Experiences of Depression

Open access
Yoo, Dong Whi, Shi, Jiayue Melissa, Rodriguez, Violeta J., Saha, Koustuv·2025·arXiv
DOI 10.48550/arxiv.2504.18932arXiv 2504.18932
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in LLMs enable chatbots to interact with individuals on a range of queries, including sensitive mental health contexts. Despite uncertainties about their effectiveness and reliability, the development of LLMs in these areas is growing, potentially leading to harms. To better identify and mitigate these harms, it is critical to understand how the values of people with lived experiences relate to the harms. In this study, we developed a technology probe, a GPT-4o based chatbot called Zenny, enabling participants to engage with depression self-management scenarios informed by previous research. We used Zenny to interview 17 individuals with lived experiences of depression. Our thematic analysis revealed key values: informational support, emotional support, personalization, privacy, and crisis management. This work explores the relationship between lived experience values, potential harms, and design recommendations for mental health AI chatbots, aiming to enhance self-management support while minimizing risks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.18932

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Interview-based experiences of users with mental health issues using LLM-CBs for mental health show that beyond positive attitudes, a wider range of users raise concerns related to data privacy, inappropriate advice, hallucinations, lack of genuine empathy, and the need for a human-in-the-loop.

    “Interview-based experiences of users with mental health issues using LLM-CBs for mental health show that beyond positive attitudes, a wider range of users raise concerns related to data privacy, inappropriate advice, hallucinations, lack of genuine empathy, and the need for a human-in-the-loop78–80.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [80]

    The Findings (pp. 8–10) report participants’ concerns about privacy, inaccurate or unsuitable advice, limited empathy, and the importance of human interaction, despite generally positive reactions to the chatbot. But the study interviewed 17 people with lived experiences of depression; it does not establish concerns across a wider range of users. Its participants called for human support, which is not the same as a human-in-the-loop system. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 129

  1. [3][3] Alessandro Acquisti, Laura Brandimarte, and George Loewenstein. 2015. Privacy and human behavior in the age of information. Science 347, 6221 (2015), 509-514. -
  2. [2][2] Jaffar Abbas. 2021. Crisis management, transnational healthcare challenges and opportunities: The intersection of COVID-19 pandemic and global mental health. Research in Globalization 3 (2021), 100037. -
  3. [4][4] Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. Large-scale analysis of counseling conversations: An application of natural language processing to mental health. TACL (2016). -
  4. [6][6] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems . 1-13. -
  5. [5][5] Awuni Prosper Mandela Amaltinga and James Fenibe Mbinta. 2020. Factors associated with depression among young people globally: a narrative review. International journal of community medicine and public health (Gujarat) 7, 9 (2020), 3711-3721. -
  6. [7][7] Katja Andrić and Atoosa Kasirzadeh. 2023. Reconciling governmental use of online targeting with democracy. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency . 1871-1881. -
  7. [9][9] Sumit Asthana, Jane Im, Zhe Chen, and Nikola Banovic. 2024. " I know even if you don't tell me": Understanding Users' Privacy Preferences Regarding AIbased Inferences of Sensitive Information for Personalization. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-21. -
  8. [10][10] Stephanie Ballard, Karen M Chappell, and Kristen Kennedy. 2019. Judgment call the game: Using value sensitive design and design fiction to surface ethical - concerns related to technology. In Proceedings of the 2019 on Designing Interactive Systems Conference . 421-433. -
  9. [12][12] Andrew BL Berry, Catherine Lim, Andrea L Hartzler, Tad Hirsch, Evette Ludman, Edward H Wagner, and James D Ralston. 2017. Eliciting values of patients with multiple chronic conditions: Evaluation of a patient-centered framework. In AMIA Annual Symposium Proceedings , Vol. 2017. American Medical Informatics Association, 430. -
  10. [13][13] Andrew BL Berry, Catherine Y Lim, Calvin A Liang, Andrea L Hartzler, Tad Hirsch, Dawn M Ferguson, Zoë A Bermet, and James D Ralston. 2021. Supporting collaborative reflection on personal values and health. Proceedings of the ACM on human-computer interaction 5, CSCW2 (2021), 1-39. -
  11. [15][15] Ananya Bhattacharjee, Joseph Jay Williams, Jonah Meyerhoff, Harsh Kumar, Alex Mariakakis, and Rachel Kornfield. 2023. Investigating the role of context in the delivery of text messages for supporting psychological wellbeing. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . -
  12. [14][14] José Manoel Bertolote, Alexandra Fleischmann, Diego De Leo, and Danuta Wasserman. 2003. Suicide and mental disorders: do we know enough? The British journal of psychiatry 183, 5 (2003), 382-383. -
  13. [16][16] Timothy W Bickmore and Rosalind W Picard. 2005. Establishing and maintaining long-term human-computer relationships. ACM Transactions on ComputerHuman Interaction (TOCHI) 12, 2 (2005), 293-327. -
  14. [18][18] Kyle Boyd, Courtney Potts, Raymond Bond, Maurice Mulvenna, Thomas Broderick, Con Burns, Andrea Bickerdike, Mike Mctear, Catrine Kostenius, Alex Vakaloudis, et al. 2022. Usability testing and trust analysis of a mental health and wellbeing chatbot. In Proceedings of the 33rd European Conference on Cognitive Ergonomics . 1-8. -
  15. [17][17] Karen Boyd. 2022. Designing up with value-sensitive design: Building a field guide for ethical ML development. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency . 2069-2082. -
  16. [19][19] Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health 11, 4 (2019), 589-597. -
  17. [21][21] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). - · arXiv
  18. [20][20] Johana Cabrera, M Soledad Loyola, Irene Magaña, and Rodrigo Rojas. 2023. Ethical dilemmas, mental health, artificial intelligence, and llm-based chatbots. In International Work-Conference on Bioinformatics and Biomedical Engineering . Springer, 313-326. -
  19. [22][22] Zhifa Chen, Yichen Lu, Mika P Nieminen, and Andrés Lucero. 2020. Creating a chatbot for and with migrants: chatbot personality drives co-design activities. In Proceedings of the 2020 acm designing interactive systems conference . 219-230. -
  20. [24][24] EunJeong Cheon and Norman Makoto Su. 2018. Futuristic autobiographies: Weaving participant narratives to elicit values around robots. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction . 388-397. -
  21. [23][23] EunJeong Cheon and Norman Makoto Su. 2016. Integrating roboticist values into a Value Sensitive Design framework for humanoid robots. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) . IEEE, 375-382. -
  22. [25][25] Dasom Choi, Sunok Lee, Sung-In Kim, Kyungah Lee, Hee Jeong Yoo, Sangsu Lee, and Hwajung Hong. 2024. Unlock Life with a Chat (GPT): Integrating Conversational AI with Large Language Models into Everyday Lives of Autistic Individuals. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-17. -
  23. [27][27] John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . 1-19. -
  24. [26][26] Shreya Chowdhary, Anna Kawakami, Mary L Gray, Jina Suh, Alexandra Olteanu, and Koustuv Saha. 2023. Can Workers Meaningfully Consent to Workplace Wellbeing Technologies?. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency . 569-582. -
  25. [28][28] Simon Coghlan, Kobi Leins, Susie Sheldrick, Marc Cheong, Piers Gooding, and Simon D'Alfonso. 2023. To chat or bot to chat: Ethical issues with using chatbots in mental health. Digital health 9 (2023), 20552076231183542. -
  26. [30][30] Reuben Crasto, Lance Dias, Dominic Miranda, and Deepali Kayande. 2021. CareBot: a mental health ChatBot. In 2021 2nd international conference for emerging technology (INCET) . IEEE, 1-5. -
  27. [29][29] Christopher Collins, Simone Arbour, Nathan Beals, Shawn Yama, Jennifer Laffier, and Zixin Zhao. 2022. Covid connect: Chat-driven anonymous story-sharing for peer support. In Proceedings of the 2022 ACM Designing Interactive Systems Conference . 301-318. -
  28. [32][32] Vedant Das Swain, Lan Gao, William A Wood, Srikruthi C Matli, Gregory D Abowd, and Munmun De Choudhury. 2023. Algorithmic Power or Punishment: Information Worker Perspectives on Passive Sensing Enabled AI Phenotyping of Performance and Wellbeing. In CHI Conference on Human Factors in Computing Systems . -
  29. [34][34] Munmun De Choudhury, Sachin R Pendse, and Neha Kumar. 2023. Benefits and harms of large language models in digital mental health. arXiv preprint arXiv:2311.14693 (2023). - · arXiv
  30. [33][33] Vedant Das Swain and Koustuv Saha. 2024. Teacher, Trainer, Counsel, Spy: How Generative AI can Bridge or Widen the Gaps in Worker-Centric Digital Phenotyping of Wellbeing. In Proceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work . 1-13. -
  31. [35][35] Kerry Dhakal. 2022. NVivo. Journal of the Medical Library Association: JMLA 110, 2 (2022), 270. -
  32. [37][37] Mateusz Dubiel, Sylvain Daronnat, and Luis A Leiva. 2022. Conversational agents trust calibration: A user-centred perspective to design. In Proceedings of the 4th Conference on Conversational User Interfaces . 1-6. -
  33. [36][36] Shipi Dhanorkar, Christine T Wolf, Kun Qian, Anbang Xu, Lucian Popa, and Yunyao Li. 2021. Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In Proceedings of the 2021 ACM Designing Interactive Systems Conference . 15911602. -
  34. [38][38] Upol Ehsan, Koustuv Saha, Munmun De Choudhury, and Mark O Riedl. 2023. Charting the sociotechnical gap in explainable ai: A framework to address the gap in xai. Proceedings of the ACM on human-computer interaction 7, CSCW1 (2023), 1-32. -
  35. [40][40] Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR mental health 4, 2 (2017), e7785. -
  36. [39][39] Annette Felgenhauer, Katharina Kaufmann, Julia Klier, and Mathias Klier. 2021. In the same boat: social support in online peer groups for career counseling. Electronic Markets 31 (2021), 197-213. -
  37. [41][41] Pin Sym Foong, Natasha Ureyang, Charisse Foo, Sajeban Antonyrex, and Gerald CH Koh. 2024. Designing for Caregiver-facing Values Elicitation Tools. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-20. -
  38. [43][43] Batya Friedman. 1996. Value-sensitive design. interactions 3, 6 (1996), 16-23. -
  39. [42][42] Ralph R Frerichs, Carol S Aneshensel, Patricia A Yokopenic, and Virginia A Clark. 1982. Physical health and depression: an epidemiologic survey. Preventive Medicine 11, 6 (1982), 639-646. -
  40. [46][46] Steven M Goodman, Erin Buehler, Patrick Clary, Andy Coenen, Aaron Donsbach, Tiffanie N Horne, Michal Lahav, Robert MacDonald, Rain Breaw Michaels, Ajit Narayanan, et al. 2022. Lampost: Design and evaluation of an ai-assisted email writing prototype for adults with dyslexia. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility . 1-18. -
  41. [45][45] Iason Gabriel. 2020. Artificial intelligence, values, and alignment. Minds and machines 30, 3 (2020), 411-437. -
  42. [47][47] Nitesh Goyal, Minsuk Chang, and Michael Terry. 2024. Designing for HumanAgent Alignment: Understanding what humans want from their agents. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems . 1-6. -
  43. [49][49] Paul A Harris, Kirstin W Scott, Laurie Lebo, NikNik Hassan, Chad Lightner, and Jill Pulley. 2012. ResearchMatch: a national registry to recruit volunteers for clinical research. Academic Medicine 87, 1 (2012), 66-73. -
  44. [48][48] MD Romael Haque and Sabirat Rubya. 2023. An overview of chatbot-based mobile mental health apps: insights from app description and user reviews. JMIR mHealth and uHealth 11, 1 (2023), e44838. -
  45. [50][50] Gillian R Hayes. 2011. The relationship of action research to human-computer interaction. ACM Transactions on Computer-Human Interaction (TOCHI) 18, 3 (2011), 1-20. -
  46. [52][52] Janie Houle, Marjolaine Gascon-Depatie, Gabrielle Bélanger-Dumontier, and Charles Cardinal. 2013. Depression self-management support: a systematic review. Patient education and counseling 91, 3 (2013), 271-279. -
  47. [53][53] Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B Bederson, Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, et al. 2003. Technology probes: inspiring design for and with families. In Proceedings of the SIGCHI conference on Human factors in computing systems . 17-24. -
  48. [55][55] Maurice Jakesch, Zana Buçinca, Saleema Amershi, and Alexandra Olteanu. 2022. How different groups prioritize ethical values for responsible AI. In Proceedings -
  49. [54][54] IEEE Spectrum. 2024. Woebot: The AI Chatbot That Wants to Help You Feel Better . https://spectrum.ieee.org/woebot Accessed: 2024-08-31. , , - of the 2022 ACM Conference on Fairness, Accountability, and Transparency . 310323. -
  50. [57][57] Eunkyung Jo, Yuin Jeong, SoHyun Park, Daniel A Epstein, and Young-Ho Kim. 2024. Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Intervention. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-21. -
  51. [56][56] Eunkyung Jo, Daniel A Epstein, Hyunhoon Jung, and Young-Ho Kim. 2023. Understanding the benefits and challenges of deploying conversational AI leveraging large language models for public health intervention. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1-16. -
  52. [58][58] Rachael M Kang and Tera L Reynolds. 2024. 'This app said I had severe depression, and now I don't know what to do': the unintentional harms of mental health applications. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-17. -
  53. [60][60] Anna Kawakami, Shreya Chowdhary, Shamsi T Iqbal, Q Vera Liao, Alexandra Olteanu, Jina Suh, and Koustuv Saha. 2023. Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders. Proceedings of the ACM on Human-Computer Interaction (CSCW) (2023). -
  54. [59][59] Harmanpreet Kaur, Eytan Adar, Eric Gilbert, and Cliff Lampe. 2022. Sensible AI: Re-imagining interpretability and explainability using sensemaking theory. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency . 702-714. -
  55. [61][61] Anna Kawakami, Venkatesh Sivaraman, Logan Stapleton, Hao-Fei Cheng, Adam Perer, Zhiwei Steven Wu, Haiyi Zhu, and Kenneth Holstein. 2022. 'Why Do I Care What's Similar?' Probing Challenges in AI-Assisted Child Welfare DecisionMaking through Worker-AI Interface Design Concepts. In Proceedings of the 2022 ACM Designing Interactive Systems Conference . 454-470. -
  56. [63][63] Meeyun Kim, Koustuv Saha, Munmun De Choudhury, and Daejin Choi. 2023. Supporters first: understanding online social support on mental health from a supporter perspective. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-28. -
  57. [62][62] Yi Xuan Khoo, Rachael M Kang, Tera L Reynolds, and Helena M Mentis. 2024. 'That's Kind of Sus (picious)': The Comprehensiveness of Mental Health Application Users' Privacy and Security Concerns. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-16. -
  58. [64][64] Taewan Kim, Seolyeong Bae, Hyun Ah Kim, Su-woo Lee, Hwajung Hong, Chanmo Yang, and Young-Ho Kim. 2024. MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-20. -
  59. [66][66] Vivian Lai, Chacha Chen, Alison Smith-Renner, Q Vera Liao, and Chenhao Tan. 2023. Towards a science of human-AI decision making: An overview of design space in empirical human-subject studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency . 1369-1385. -
  60. [65][65] Tin Lai, Yukun Shi, Zicong Du, Jiajie Wu, Ken Fu, Yichao Dou, and Ziqi Wang. 2023. Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Models. arXiv:2307.11991 [cs.CL] - · arXiv
  61. [67][67] Lauren Lamberton, John Devaney, and Lisa Bunting. 2016. New challenges in family support: the use of digital technology in supporting parents. Child abuse review 25, 5 (2016), 359-372. -
  62. [69][69] Christopher A Le Dantec, Erika Shehan Poole, and Susan P Wyche. 2009. Values as lived experience: evolving value sensitive design in support of value discovery. In Proceedings of the SIGCHI conference on human factors in computing systems . -
  63. [68][68] Hannah R Lawrence, Renee A Schneider, Susan B Rubin, Maja J Matarić, Daniel J McDuff, and Megan Jones Bell. 2024. The opportunities and risks of large language models in mental health. JMIR Mental Health 11, 1 (2024), e59479. -
  64. [70][70] Reeva Lederman, John Gleeson, Greg Wadley, Simon D'alfonso, Simon Rice, Olga Santesteban-Echarri, and Mario Alvarez-Jimenez. 2019. Support for carers of young people with mental illness: design and trial of a technology-mediated therapy. ACM Transactions on Computer-Human Interaction (TOCHI) (2019). -
  65. [72][72] Yi-Chieh Lee, Naomi Yamashita, and Yun Huang. 2020. Designing a chatbot as a mediator for promoting deep self-disclosure to a real mental health professional. Proceedings of the ACM on Human-Computer Interaction 4, CSCW1 (2020), 1-27. -
  66. [71][71] Hao-Ping Lee, Yu-Ju Yang, Thomas Serban Von Davier, Jodi Forlizzi, and Sauvik Das. 2024. Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Risks. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-19. -
  67. [73][73] Yi-Chieh Lee, Naomi Yamashita, Yun Huang, and Wai Fu. 2020. " I hear you, I feel you": encouraging deep self-disclosure through a chatbot. In Proceedings of the 2020 CHI conference on human factors in computing systems . 1-12. -
  68. [75][75] Tianshi Li, Sauvik Das, Hao-Ping Lee, Dakuo Wang, Bingsheng Yao, and Zhiping Zhang. 2024. Human-Centered Privacy Research in the Age of Large Language Models. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems . 1-4. -
  69. [74][74] Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F Chen, Zhengyuan Liu, and Diyi Yang. 2023. Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. arXiv preprint arXiv:2310.15638 (2023). - · arXiv
  70. [76][76] QVera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-15. -
  71. [78][78] Catherine Y Lim, Andrew BL Berry, Tad Hirsch, Andrea L Hartzler, Edward H Wagner, Evette J Ludman, and James D Ralston. 2017. Understanding what is most important to individuals with multiple chronic conditions: a qualitative study of patients' perspectives. Journal of general internal medicine 32 (2017), 1278-1284. -
  72. [77][77] Catherine Y Lim, Andrew BL Berry, Andrea L Hartzler, Tad Hirsch, David S Carrell, Zoë A Bermet, and James D Ralston. 2019. Facilitating self-reflection about values and self-care among individuals with chronic conditions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . 1-12. -
  73. [79][79] Shuai Ma, Ying Lei, Xinru Wang, Chengbo Zheng, Chuhan Shi, Ming Yin, and Xiaojuan Ma. 2023. Who should i trust: Ai or myself? leveraging human and ai correctness likelihood to promote appropriate trust in ai-assisted decisionmaking. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1-19. -
  74. [81][81] Zilin Ma, Yiyang Mei, and Zhaoyuan Su. 2023. Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support. In AMIA Annual Symposium Proceedings , Vol. 2023. American Medical Informatics Association, 1105. -
  75. [80][80] Zilin Ma, Yiyang Mei, Yinru Long, Zhaoyuan Su, and Krzysztof Z Gajos. 2024. Evaluating the Experience of LGBTQ+ People Using Large Language Model Based Chatbots for Mental Health Support. arXiv preprint arXiv:2402.09260 (2024). - · arXiv
  76. [82][82] Michael A Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-14. -
  77. [83][83] Lena Mamykina, Daniel A. Epstein, Predrag Klasnja, Donna Sprujt-Metz, Jochen Meyer, Mary Czerwinski, Tim Althoff, Eun Kyoung Choe, Munmun De Choudhury, and Brian Lim. 2022. Grand challenges for personal informatics and AI. In CHI Conference on Human Factors in Computing Systems Extended Abstracts . 1-6. -
  78. [85][85] Elliot Mitchell, Noemie Elhadad, and Lena Mamykina. 2022. Examining AI methods for micro-coaching dialogs. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . 1-24. -
  79. [87][87] Subigya Nepal, Arvind Pillai, Weichen Wang, Tess Griffin, Amanda C Collins, Michael Heinz, Damien Lekkas, Shayan Mirjafari, Matthew Nemesure, George Price, et al. 2024. MoodCapture: Depression Detection Using In-the-Wild Smartphone Images. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-18. -
  80. [88][88] Shi Hui Ng, Lay-Ki Soon, and Tin Tin Su. 2023. Emotion-Aware Chatbot with Cultural Adaptation for Mitigating Work-Related Stress. In Proceedings of the Asian HCI Symposium 2023 . 41-50. -
  81. [90][90] William Odom, John Zimmerman, Scott Davidoff, Jodi Forlizzi, Anind K Dey, and Min Kyung Lee. 2012. A fieldwork of the future with user enactments. In Proceedings of the Designing Interactive Systems Conference . 338-347. -
  82. [89][89] Robert Nimmo, Marios Constantinides, Ke Zhou, Daniele Quercia, and Simone Stumpf. 2024. User Characteristics in Explainable AI: The Rabbit Hole of Personalization?. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-13. -
  83. [93][93] Harshvardhan J Pandit and Dave Lewis. 2018. Ease and ethics of user profiling in black mirror. In Companion Proceedings of the The Web Conference 2018 . 15771583. -
  84. [94][94] SoHyun Park, Anja Thieme, Jeongyun Han, Sungwoo Lee, Wonjong Rhee, and Bongwon Suh. 2021. 'I wrote as if I were telling a story to someone I knew.': Designing Chatbot Interactions for Expressive Writing in Mental Health. In Proceedings of the 2021 ACM Designing Interactive Systems Conference . 926-941. -
  85. [95][95] Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Christopher Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) . 9308-9319. -
  86. [97][97] Kat Roemmich, Florian Schaub, and Nazanin Andalibi. 2023. Emotion AI at work: Implications for workplace surveillance, emotional labor, and emotional privacy. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1-20. -
  87. [98][98] Yasaman Rohanifar, Sharifa Sultana, Shaid Hasan, Priyank Chandra, and Syed Ishtiaque Ahmed. 2022. " Kabootar": Towards Informal, Trustworthy, and Community-Based FinTech for Marginalized Immigrants. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1-32. -
  88. [100][100] Hyeyoung Ryu, Andrew BL Berry, Catherine Y Lim, Andrea Hartzler, Tad Hirsch, Juanita I Trejo, Zoë Abigail Bermet, Brandi Crawford-Gallagher, Vi Tran, Dawn Ferguson, et al. 2023. 'You Can See the Connections': Facilitating Visualization of Care Priorities in People Living with Multiple Chronic Health Conditions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . -
  89. [99][99] Jonathan Rottenberg, James J Gross, and Ian H Gotlib. 2005. Emotion context insensitivity in major depressive disorder. Journal of abnormal psychology 114, 4 (2005), 627. -
  90. [101][101] Sahand Sabour, Wen Zhang, Xiyao Xiao, Yuwei Zhang, Yinhe Zheng, Jiaxin Wen, Jialu Zhao, and Minlie Huang. 2023. A chatbot for mental health support: exploring the impact of Emohaa on reducing mental distress in China. Frontiers in digital health 5 (2023), 1133987. -
  91. [103][103] Koustuv Saha, Yoshee Jain, Chunyu Liu, Sidharth Kaliappan, and Ravi Karkar. 2025. AI vs. Humans for Online Support: Comparing the Language of Responses from LLMs and Online Communities of Alzheimer's Disease. ACM Transactions on Computing for Healthcare (2025). -
  92. [102][102] Malak Sadek, Marios Constantinides, Daniele Quercia, and Céline Mougenot. 2024. Guidelines for Integrating Value Sensitive Design in Responsible AI Toolkits. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-20. -
  93. [106][106] Corina Sas, Kobi Hartley, and Muhammad Umair. 2020. ManneqKit cards: A kinesthetic empathic design tool communicating depression experiences. In Proceedings of the 2020 acm designing interactive systems conference . 1479-1493. -
  94. [105][105] Ziggi Ivan Santini, Ai Koyanagi, Stefanos Tyrovolas, Catherine Mason, and Josep Maria Haro. 2015. The association between social relationships and depression: A systematic review. Journal of affective disorders 175 (2015), 53-65. -
  95. [107][107] Corina Sas, Kristina Höök, Gavin Doherty, Pedro Sanches, Tim Leufkens, and Joyce Westerink. 2020. Mental wellbeing: future agenda drawing from design, HCI and big data. In Companion publication of the 2020 ACM designing interactive systems conference . 425-428. -
  96. [109][109] Eva Sharma and Munmun De Choudhury. 2018. Mental health support and its relationship to linguistic accommodation in online communities. In Proceedings of the 2018 CHI conference on human factors in computing systems . 1-13. -
  97. [108][108] Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, Theresa Nguyen, and Tim Althoff. 2024. Facilitating self-guided mental health interventions through human-language model interaction: A case study of cognitive restructuring. In Proceedings of the CHI Conference on Human Factors in Computing Systems . -
  98. [110][110] Hua Shen, Tiffany Knearem, Reshmi Ghosh, Kenan Alkiek, Kundan Krishna, Yachuan Liu, Ziqiao Ma, Savvas Petridis, Yi-Hao Peng, Li Qiwei, et al. 2024. Towards Bidirectional Human-AI Alignment: A Systematic Review for Clarifications, Framework, and Future Directions. arXiv preprint arXiv:2406.09264 (2024). - · arXiv
  99. [112][112] Petr Slovak, Alissa Antle, Nikki Theofanopoulou, Claudia Daudén Roquet, James Gross, and Katherine Isbister. 2023. Designing for emotion regulation interventions: an agenda for HCI theory and research. ACM Transactions on ComputerHuman Interaction 30, 1 (2023), 1-51. -
  100. [111][111] Kellie Yu Hui Sim, Kohleen Tijing Fortuno, and Kenny Tsu Wei Choo. 2024. Towards Understanding Emotions for Engaged Mental Health Conversations. In Companion Publication of the 2024 ACM Designing Interactive Systems Conference . 176-180. -
  101. [113][113] Amalie Søgaard Neilsen and Rhonda L Wilson. 2019. Combining e-mental health intervention development with human computer interaction (HCI) design to enhance technology-facilitated recovery for people with depression and/or anxiety conditions: An integrative literature review. International journal of mental health nursing 28, 1 (2019), 22-39. -
  102. [115][115] Velvet Spors, Hanne Gesine Wagner, Martin Flintham, Pat Brundell, and David Murphy. 2021. Selling glossy, easy futures: A feminist exploration of commercial mental-health-focused self-care apps' descriptions in the google play store. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1-17. -
  103. [114][114] Inhwa Song, Sachin R Pendse, Neha Kumar, and Munmun De Choudhury. 2024. The typing cure: Experiences with large language model chatbots for mental health support. arXiv preprint arXiv:2401.14362 (2024). - · arXiv
  104. [116][116] Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-26. -
  105. [117][117] Melissa Suran. 2023. How the new 988 lifeline is helping millions in mental health crisis. JAMA (2023). , , Perceptions and views from mental healthcare professionals and experts. ACM Transactions on Computing for Healthcare 2, 3 (2021), 1-15. -
  106. [120][120] Yuan-Chi Tseng, Weerachaya Jarupreechachan, and Tuan-He Lee. 2023. Understanding the benefits and design of chatbots to meet the healthcare needs of migrant workers. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1-34. -
  107. [119][119] Michael Terry, Chinmay Kulkarni, Martin Wattenberg, Lucas Dixon, and Meredith Ringel Morris. 2023. AI alignment in the design of interactive AI: Specification alignment, process alignment, and evaluation support. arXiv preprint arXiv:2311.00710 (2023). - · arXiv
  108. [121][121] Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S Kashavan, and John Blake Torous. 2019. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. The Canadian Journal of Psychiatry 64, 7 (2019), 456-464. -
  109. [123][123] Roos A van Grieken, Anneloes CE Kirkenier, Maarten WJ Koeter, Udo W Nabitz, and Aart H Schene. 2015. Patients' perspective on self-management in the recovery from depression. Health Expectations 18, 5 (2015), 1339-1348. -
  110. [122][122] Niels van Berkel, Jorge Goncalves, Lauri Lovén, Denzil Ferreira, Simo Hosio, and Vassilis Kostakos. 2019. Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports. International Journal of HumanComputer Studies 125 (2019), 118-128. -
  111. [124][124] Rosa A van Grieken, Mirjam J van Tricht, Maarten WJ Koeter, Wim van den Brink, and Aart H Schene. 2018. The use and helpfulness of self-management strategies for depression: the experiences of patients. PLoS One 13, 10 (2018), e0206262. -
  112. [126][126] Yi-Chia Wang, Robert Kraut, and John M Levine. 2012. To stay or leave? The relationship of emotional and informational support to commitment in online health support groups. In Proceedings of the ACM 2012 conference on computer supported cooperative work . 833-842. -
  113. [125][125] Qiaosi Wang, Shan Jing, and Ashok K Goel. 2022. Co-designing AI agents to support social connectedness among online learners: functionalities, social characteristics, and ethical challenges. In Proceedings of the 2022 ACM Designing Interactive Systems Conference . 541-556. -
  114. [127][127] Justin D Weisz, Jessica He, Michael Muller, Gabriela Hoefer, Rachel Miles, and Werner Geyer. 2024. Design Principles for Generative AI Applications. In Proceedings of the CHI Conference on Human Factors in Computing Systems . -
  115. [129][129] Langdon Winner. 1978. Autonomous technology: Technics-out-of-control as a theme in political thought . Mit Press. -
  116. [128][128] Joel Wester, Henning Pohl, Simo Hosio, and Niels van Berkel. 2024. " This Chatbot Would Never...": Perceived Moral Agency of Mental Health Chatbots. Proceedings of the ACM on Human-Computer Interaction 8, CSCW1 (2024), 1-28. -
  117. [130][130] Langdon Winner. 2017. Do artifacts have politics? In Computer ethics . Routledge. -
  118. [132][132] Lu Xu, Leslie Sanders, Kay Li, James CL Chow, et al. 2021. Chatbot for health care and oncology applications using artificial intelligence and machine learning: systematic review. JMIR cancer 7, 4 (2021), e27850. -
  119. [131][131] Ling Wu, Joshua Paolo Seguin, Dharshani Chandrasekara, Mairead Claire Cardamone-Breen, Jue Xie, Roisin Mcnaney, Tom Bartindale, Patrick Olivier, and Marie BH Yap. 2024. Designing online peer support for parents of adolescents at risk of mental health challenges. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-14. -
  120. [133][133] Wei Xu, Marvin J Dainoff, Liezhong Ge, and Zaifeng Gao. 2023. Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI. International Journal of HumanComputer Interaction 39, 3 (2023), 494-518. -
  121. [134][134] Qian Yang, Richmond Y Wong, Steven Jackson, Sabine Junginger, Margaret D Hagan, Thomas Gilbert, and John Zimmerman. 2024. The Future of HCI-Policy Collaboration. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-15. -
  122. [136][136] Daisy Yoo. 2017. Stakeholder tokens: A constructive method for value sensitive design stakeholder analysis. In Proceedings of the 2017 ACM conference companion publication on designing interactive systems . 280-284. -
  123. [138][138] Dong Whi Yoo, Michael L Birnbaum, Anna R Van Meter, Asra F Ali, Elizabeth Arenare, Gregory D Abowd, and Munmun De Choudhury. 2020. Designing a Clinician-Facing Tool for Using Insights From Patients' Social Media Activity: Iterative Co-Design Approach. JMIR Mental Health 7, 8 (2020), e16969. -
  124. [137][137] Dong Whi Yoo, Aditi Bhatnagar, Sindhu Kiranmai Ernala, Asra Ali, Michael L Birnbaum, Gregory D Abowd, and Munmun De Choudhury. 2023. Discussing social media during psychotherapy consultations: patient narratives and privacy implications. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-24. -
  125. [139][139] Dong Whi Yoo, Hayoung Woo, Sachin R Pendse, Nathaniel Young Lu, Michael L Birnbaum, Gregory D Abowd, and Munmun De Choudhury. 2024. Missed Opportunities for Human-Centered AI Research: Understanding Stakeholder Collaboration in Mental Health AI Research. Proceedings of the ACM on HumanComputer Interaction 8, CSCW1 (2024), 1-24. -
  126. [140][140] Nima Zargham, Dmitry Alexandrovsky, Jan Erich, Nina Wenig, and Rainer Malaka. 2022. 'I Want It That Way': Exploring Users' Customization and Personalization Preferences for Home Assistants. In CHI conference on human factors in computing systems extended abstracts . 1-8. -
  127. [141][141] Jiawei Zhou, Koustuv Saha, Irene Michelle Lopez Carron, Dong Whi Yoo, Catherine R Deeter, Munmun De Choudhury, and Rosa I Arriaga. 2022. Veteran Critical Theory as a Lens to Understand Veterans' Needs and Support on Social Media. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1-28. -
  128. [142][142] Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023. Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . -
  129. [143][143] Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018. Value-sensitive algorithm design: Method, case study, and lessons. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1-23.

Cited by

Sorted by citations

Authors
Yoo, D. W., Shi, J. M., Rodriguez, V. J., & Saha, K.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.18932
arXiv
2504.18932
OpenAlex
W4416891275
Cited by
2
References
0
Citations by year
2026: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
mental health chatbots, depression, lived experience, values, harm, self-management, emotional support, information support, personalization, privacy, crisis management, GPT-4o, technology probe

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review