CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy
Open accessAbstract
There is a significant gap between patient needs and available mental health support today. In this paper, we aim to thoroughly examine the potential of using Large Language Models (LLMs) to assist professional psychotherapy. To this end, we propose a new benchmark, CBT-BENCH, for the systematic evaluation of cognitive behavioral therapy (CBT) assistance. We include three levels of tasks in CBT-BENCH: I: Basic CBT knowledge acquisition, with the task of multiple-choice questions; II: Cognitive model understanding, with the tasks of cognitive distortion classification, primary core belief classification, and fine-grained core belief classification; III: Therapeutic response generation, with the task of generating responses to patient speech in CBT therapy sessions. These tasks encompass key aspects of CBT that could potentially be enhanced through AI assistance, while also outlining a hierarchy of capability requirements, ranging from basic knowledge recitation to engaging in real therapeutic conversations. We evaluated representative LLMs on our benchmark. Experimental results indicate that while LLMs perform well in reciting CBT knowledge, they fall short in complex real-world scenarios requiring deep analysis of patients' cognitive structures and generating effective responses, suggesting potential future work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Rigorous comparisons of LLMs-as-therapists cast doubt on their current performance.
“Rigorous comparisons of LLMs-as-therapists cast doubt on their current performance.”
From Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. (2025) · cites this paper as [156]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Wang, R., Milani, S., Chiu, J. C., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2405.19660 · Cited by 31 · see citing works
Self-citationHelping the Helper: Supporting Peer Counselors via AI-Empowered Practice and FeedbackPreprint
Hsu, S.-L., Shah, R. S., Senthil, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.08982 · Cited by 21 · see citing works
Flagged first, then by citations
Research works printed but not linked: 24
- Aaron T Beck and Emily AP Haigh. 2014. Advances in cognitive theory and therapy: The generic cognitive model. Annual review of clinical psychology, 10(1):1- 24.
- Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023a. LLM- empowered chatbots for psychiatrist and patient sim- ulation: Application and evaluation. arXiv [cs.CL].
- Zhiyu Chen, Yujie Lu, and William Wang. 2023b. Em- powering psychotherapy with large language models: Cognitive distortion detection through diagnosis of thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4295-4304.
- Shuang Gao, Vince D Calhoun, and Jing Sui. 2018. Machine learning in major depression: From classifi- cation to treatment outcome prediction. CNS neuro- science & therapeutics, 24(11):1037-1052.
- Gemma Team. 2024. Gemma 2: Improving open lan- guage models at a practical size. arXiv [cs.CL].
- Declan Grabb, Max Lamparth, and Nina Vasan. 2024. Risks from language models for automated mental healthcare: Ethics and structure for implementation. arXiv [cs.CY].
- Shaoxiong Ji, Tianlin Zhang, Kailai Yang, Sophia Ana- niadou, and Erik Cambria. 2023. Rethinking large language models in mental health applications. arXiv [cs.CL].
- Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv [cs.CL].
- Dae-Young Kim, Rebecca Hwa, and Muhammad Mah- bubur Rahman. 2024. MhGPT: A lightweight gener- ative pre-trained transformer for mental health text analysis. arXiv [cs.CL].
- Willem Kuyken, Christine A Padesky, and Robert Dudley. 2011. Collaborative case conceptualiza- tion: Working effectively with clients in cognitive- behavioral therapy. Guilford Press.
- June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023. ChatCounselor: A large language models for mental health support. arXiv [cs.CL].
- Llama Team. 2024. The llama 3 herd of models. arXiv [cs.AI].
- Mounica Maddela, Megan Ung, Jing Xu, Andrea Madotto, Heather Foran, and Y-Lan Boureau. 2023. Training models to generate, recognize, and reframe unhelpful thoughts. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Stroudsburg, PA, USA. Association for Computational Linguistics.
- Saskia Senn, ML Tlachac, Ricardo Flores, and Elke Rundensteiner. 2022. Ensembles of bert for de- pression classification. In 2022 44th Annual In- ternational Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 4691- 4694. IEEE.
- Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. 2021. Towards facilitating empathic conversations in online mental health sup- port: A reinforcement learning approach. In Proceed- ings of the Web Conference 2021, pages 194-205.
- Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. 2023a. Human-ai collabo- ration enables more empathic conversations in text- based peer-to-peer mental health support. Nature Machine Intelligence, 5(1):46-57.
- Ashish Sharma, Kevin Rushton, Inna Lin, David Wad- den, Khendra Lucas, Adam Miner, Theresa Nguyen, and Tim Althoff. 2023b. Cognitive reframing of negative thoughts through human-language model interaction. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), Stroudsburg, PA, USA. Association for Computational Linguistics.
- Ashish Sharma, Kevin Rushton, Inna Lin, David Wad- den, Khendra Lucas, Adam Miner, Theresa Nguyen, and Tim Althoff. 2023c. Cognitive reframing of neg- ative thoughts through human-language model inter- action. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 9977-10000.
- Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, Theresa Nguyen, and Tim Althoff. 2024. Facilitat- ing self-guided mental health interventions through human-language model interaction: A case study of cognitive restructuring. In Proceedings of the CHI Conference on Human Factors in Computing Systems, volume 21, pages 1-29, New York, NY, USA. ACM.
- Sagarika Shreevastava and Peter Foltz. 2021a. Detect- ing cognitive distortions from patient-therapist inter- actions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, Stroudsburg, PA, USA. Associa- tion for Computational Linguistics.
- Sagarika Shreevastava and Peter Foltz. 2021b. Detect- ing cognitive distortions from patient-therapist inter- actions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 151-158.
- Bichen Wang, Pengfei Deng, Yanyan Zhao, and Bing Qin. 2023. C2D2 dataset: A resource for the cog- nitive distortion analysis and its impact on mental health. In Findings of the Association for Compu- tational Linguistics: EMNLP 2023, pages 10149- 10160, Stroudsburg, PA, USA. Association for Com- putational Linguistics.
- Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. To- wards interpretable mental health analysis with large language models. arXiv [cs.CL].
- Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. MentaL- LaMA: Interpretable mental health analysis on social media with large language models. In Proceedings of the ACM on Web Conference 2024, volume 35 of WWW '24, page 4489-4500. ACM.
Cited by
Sorted by citations
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204 · see citing works
Do Large Language Models Align with Core Mental Health Counseling Competencies?
Nguyen, V. C., Taher, M., Hong, D., Possobom, V. K., Gopalakrishnan, V. T., Raj, E., Li, Z., Soled, H. J., Birnbaum, M. L., Kumar, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.22446 · Cited by 3 · see citing works
- Authors
- Zhang, M., Yang, X., Zhang, X., Labrum, T., Chiu, J. C., Eack, S. M., Fang, F., Wang, W. Y., & Chen, Z. Z.
- Affiliations
- The University of Texas at Dallas (US)
- University of California, Santa Barbara (US)
- University of Pittsburgh (US)
- Princeton University (US)
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2410.13218
- OpenAlex
- W4403579420
- Cited by
- 2
- References
- 2
- Citations by year
- 2026: 1 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- large language models, cognitive behavioral therapy, benchmark, AI-assisted psychotherapy, mental health support, LLM evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
