CoChat
Sign Up
PreprintCited by 6Digital Mental Health InterventionsSocial Robot Interaction and HRIPsychotherapy Techniques and Applications

Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions

Open access
Iftikhar, Zainab, Ransom, Sean, Xiao, Amy, Nugent, Nicole, Huang, Jeff·2024·arXiv
DOI 10.48550/arxiv.2409.02244arXiv 2409.02244
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) are increasingly being used as ad hoc therapists. While prior research has found that LLMs outperform human counselors in generating single-turn empathetic responses, fewer studies have compared their behaviors across multi-turn sessions. In this study, we compare the session-level behaviors of human peer counselors with those of an LLM, both trained on the same manual to deliver multi-turn, single-session Cognitive Behavioral Therapy (CBT). Our three-phase, mixed-methods study involved: (a) an 18-month ethnography of a peer support platform, where seven counselors iteratively refined CBT prompts through 110 self-counseling sessions and 60 weekly focus groups; (b) a novel session generation method that allows direct, controlled comparison of human and LLM counselors under matched conditions---client responses were drawn from publicly available human-led CBT sessions while counselor responses were generated by a CBT-prompted LLM; and (c) expert evaluations conducted by three licensed clinical psychologists. Through data triangulation, our results show a trade-off. Human peer counselors use relational techniques to interpret subtle cues, adapt CBT to users' values and cultural contexts, and use strategies such as small talk and contextually relevant self-disclosure to build rapport and guide the session, but often at the expense of session structure and therapeutic focus. LLM counselors, on the other hand, demonstrate greater methodological adherence to CBT techniques, but struggle to sustain turn-taking, frequently fail to distinguish between clinically important and trivial content, and are more prone to lecturing and imposing solutions. LLM counselors also tend to produce ``deceptive empathy'', excessively anthropomorphic responses that can inflate user expectations of genuine human care.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2409.02244

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. Aaron T. Beck. 1976. Cognitive Therapy and the Emotional Disorders. International Universities Press, New York.
  2. Timothy Bickmore and Justine Cassell. 2001. Relational agents: a model and implementation of building user trust. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI '01). 396-403. https://doi.org/10.1145/365024.365304 · DOI · Paper page
  3. Thomas D Borkovec, Michelle G Newman, Aaron L Pincus, and Richard Lytle. 2002. A Component Analysis of Cognitive-Behavioral Therapy for Generalized Anxiety Disorder and the Role of Interpersonal Problems. Journal of Consulting and Clinical Psychology 70, 2 (2002), 288-298.
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS '20). Article 159, 25 pages.
  5. Linda Campbell, Melba Vasquez, Stephen Behnke, and Robert Kinscherff. 2010. APA Ethics Code Commentary and Case Illustrations. American Psychological Association, Washington, D.C. http://www.jstor.org/stable/j.ctv1chs5gx
  6. Domenic V Cicchetti. 1994. Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. Psychological Assessment 6, 4 (1994), 284-290.
  7. Mick Cooper and John McLeod. 2007. A pluralistic framework for counselling and psychotherapy: Implications for research. Counselling and Psychotherapy Research 7, 3 (2007), 135-143.
  8. Windy Dryden. 2022. Single-Session Integrated CBT: Distinctive Features (2 ed.). Routledge, London. https://doi.org/10.4324/9781003214557 · DOI
  9. Sindhu Kiranmai Ernala, Michael L. Birnbaum, Kristin A. Candan, Asra F. Rizvi, William A. Sterling, John M. Kane, and Munmun De Choudhury. 2019. Methodological Gaps in Predicting Mental Health States from Social Media: Triangulating Diagnostic Signals. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). 1-16. https://doi.org/10.1145/3290605.3300364 · DOI
  10. Barney Glaser and Anselm Strauss. 1999. Discovery of Grounded Theory: Strategies for Qualitative Research (1st ed.). Routledge, New York. https://doi.org/10.4324/9780203793206 · DOI · Paper page
  11. Barney G Glaser. 1965. The Constant Comparative Method of Qualitative Analysis. Social Problems 12, 4 (1965), 436-445. http://www.jstor.org/stable/798843
  12. Ingrid D Goldstrom, Jean Campbell, Joseph A Rogers, David B Lambert, Beatrice Blacklow, Marilyn J Henderson, and Ronald W Manderscheid. 2006. National Estimates for Mental Health Mutual Support Groups, Self-Help Organizations, and Consumer-Operated Services. Administration and Policy in Mental Health and Mental Health Services Research 33 (2006), 92-103.
  13. Ziyin Gu and Qingmeng Zhu. 2023. MentalBlend: Enhancing Online Mental Health Support through the Integration of LLMs with Psychological Counseling Theories. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 46.
  14. Michael V Heinz, Daniel M Mackin, Brianna M Trudeau, Sukanya Bhattacharya, Yinzhou Wang, Haley A Banta, Abi D Jewett, Abigail J Salzhauer, Tess Z Griffin, and Nicholas C Jacobson. 2025. Randomized Trial of a Generative AI Chatbot for Mental Health Treatment. NEJM AI 2, 4 (2025), AIoa2400802.
  15. Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022. MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. 7184-7190. https://aclanthology.org/2022.lrec-1.778/
  16. Terry K Koo and Mae Y Li. 2016. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine 15, 2 (2016), 155-163.
  17. Zilin Ma, Yiyang Mei, and Zhaoyuan Su. 2024. Understanding the Benefits and Challenges of Using Large Language Model-based Conversational Agents for Mental Well-being Support. AMIA Annual Symposium Proceedings 2023 (January 2024), 1105-1114. https://pmc.ncbi.nlm.nih.gov/articles/PMC10785945/
  18. Shery Mead and Cheryl MacNeil. 2006. Peer Support: What Makes It Unique? International Journal of Psychosocial Rehabilitation 10, 2 (2006), 29-37.
  19. Hongbin Na. 2024. CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 2930-2940. https://aclanthology.org/2024.lrec-main.261/
  20. Yada Pruksachatkun, Sachin R. Pendse, and Amit Sharma. 2019. Moments of Change: Analyzing Peer-Based Cognitive Support in Online Mental Health Forums. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). 1-13. https://doi.org/10.1145/3290605.3300294 · DOI
  21. Stefan Rennick-Egglestone, Sarah Knowles, Gill Toms, Penny Bee, Karina Lovell, and Peter Bower. 2016. Health Technologies 'In the Wild': Experiences of Engagement with Computerised CBT. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). 2124-2135. https://doi.org/10.1145/2858036.2858128 · DOI
  22. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. Towards understanding sycophancy in language models. In 12th International Conference on Learning Representations, ICLR 2024.
  23. Patrick E Shrout and Joseph L Fleiss. 1979. Intraclass Correlations: Uses in Assessing Rater Reliability. Psychological Bulletin 86, 2 (1979), 420.
  24. Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. 2022. Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (Los Angeles, CA, USA) (CCS '22). 2659-2673. https://doi.org/10.1145/3548606.3560599 · DOI
  25. Xin Sun, Xiao Tang, Abdallah El Ali, Zhuying Li, Pengjie Ren, Jan de Wit, Jiahuan Pei, and Jos A Bosch. 2025. Rethinking the Alignment of Psychotherapy Dialogue Generation with Motivational Interviewing Strategies. In Proceedings of the 31st International Conference on Computational Linguistics. 1983-2002. https://aclanthology.org/2025.coling-main.136/
  26. Yi-Chia Wang, Robert Kraut, and John M. Levine. 2012. To stay or leave? the relationship of emotional and informational support to commitment in online health support groups. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (Seattle, Washington, USA) (CSCW '12). 833-842. https://doi.org/10.1145/2145204.2145329 · DOI · Paper page
  27. Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghassemi, Anind K. Dey, and Dakuo Wang. 2024. Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, 1, Article 31 (March 2024), 32 pages. https://doi.org/10.1145/3643540 · DOI · Paper page
  28. Svetlana Yarosh. 2013. Shifting dynamics or breaking sacred traditions? the role of technology in twelve-step fellowships. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). 3413-3422. https://doi.org/10.1145/2470654.2466468 · DOI

Cited by

Sorted by citations

Related

Authors
Iftikhar, Z., Ransom, S., Xiao, A., Nugent, N., & Huang, J.
Affiliations
  • Brown University (US)
  • Louisiana State University (US)
  • University of Washington (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
10
Issue
6
Pages
1-34
ISSN
2573-0142 (ISSN-L)
DOI
10.48550/arxiv.2409.02244
arXiv
2409.02244
OpenAlex
W7214082659
Cited by
6
References
56
Citations by year
2026: 4 · 2025: 2
FWCI
23.66
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
Keywords
cognitive behavioral therapy, peer counseling, human-AI comparison, anthropomorphism, turn-taking, rapport building, self-disclosure, session structure, AI ethics, hybrid human-AI systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review