Do Large Language Models Align with Core Mental Health Counseling Competencies?
Open accessAbstract
The rapid evolution of Large Language Models (LLMs) presents a promising solution to the global shortage of mental health professionals. However, their alignment with essential counseling competencies remains underexplored. We introduce CounselingBench, a novel NCMHCE-based benchmark evaluating 22 general-purpose and medical-finetuned LLMs across five key competencies. While frontier models surpass minimum aptitude thresholds, they fall short of expert-level performance, excelling in Intake, Assessment & Diagnosis but struggling with Core Counseling Attributes and Professional Practice & Ethics. Surprisingly, medical LLMs do not outperform generalist models in accuracy, though they provide slightly better justifications while making more context-related errors. These findings highlight the challenges of developing AI for mental health counseling, particularly in competencies requiring empathy and nuanced reasoning. Our results underscore the need for specialized, fine-tuned models aligned with core mental health counseling competencies and supported by human oversight before real-world deployment. Code and data associated with this manuscript can be found at: https://github.com/cuongnguyenx/CounselingBench
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Rigorous comparisons of LLMs-as-therapists cast doubt on their current performance.
“Rigorous comparisons of LLMs-as-therapists cast doubt on their current performance.”
From Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. (2025) · cites this paper as [104]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · see citing works
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04132 · Cited by 118 · cited 1×
“Finally, for benchmarking against state-of-the-art proprietary systems, we include gpt-4o-2024-08-06 in our comparison (Bubeck et al., 2023; Achiam et al., 2023) due to its top performance on LLM leaderboards such as Chatbot Arena (Chiang et al., 2024).”
Zhou, X., Su, Z., Eisape, T., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2403.05020 · Cited by 82 · see citing works
Evaluation of ChatGPT for NLP-based Mental Health ApplicationsPreprint
Lamichhane, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.15727 · Cited by 55 · see citing works
Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language ModelsPreprint
Lai, T., Shi, Y., Du, Z., Wu, J., Fu, K., Dou, Y., & Wang, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.11991 · Cited by 39 · see citing works
Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Toma, A., Lawler, P. R., Ba, J., Krishnan, R. G., Rubin, B. B., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2305.12031 · Cited by 36 · see citing works
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.12022 · Cited by 30 · see citing works
Inverse Scaling: When Bigger Isn't Better
McKenzie, I. R., Lyzhov, A., Pieler, M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Kirtland, A., Ross, A., Liu, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.09479 · Cited by 29 · see citing works
A Computational Framework for Behavioral Assessment of LLM Therapists
Chiu, Y. Y., Sharma, A., Lin, I. W., & Althoff, T. (2024) · arXiv · doi:10.48550/arxiv.2401.00820 · Cited by 27 · see citing works
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
Anwar, U., Saparov, A., Rando, J., Paleka, D., Turpin, M., Hase, P., Lubana, E. S., Jenner, E., Casper, S., Sourbut, O., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.09932 · Cited by 19 · see citing works
Christophe, C., Kanithi, P. K., Munjal, P., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.14779 · Cited by 14 · see citing works
Chen, Z., Lu, Y., & Wang, W. Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07146 · Cited by 4 · see citing works
CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy
Zhang, M., Yang, X., Zhang, X., Labrum, T., Chiu, J. C., Eack, S. M., Fang, F., Wang, W. Y., & Chen, Z. Z. (2024) · arXiv · doi:10.48550/arxiv.2410.13218 · Cited by 2 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 18
- Paul G Banikiotes. 1977. The training of counseling psychologists. The Counseling Psychologist, 7(2):23- 26.
- Avishek Choudhury and Zaira Chaudhry. 2024. Large language models and user trust: Consequence of self- referential learning loop and the deskilling of health care professionals. Journal of Medical Internet Re- search, 26:e56764.
- Carla Clasen, Cheryl Meyer, Carl Brun, William Mase, and Kate Cauley. 2003. Development of the compe- tency assessment tool-mental health, an instrument to assess core competencies for mental health care work- ers. Psychiatric Rehabilitation Journal, 27(1):10.
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serv- ing with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626.
- Hannah R Lawrence, Renee A Schneider, Susan B Rubin, Maja J Matarić, Daniel J McDuff, and Megan Jones Bell. 2024. The opportunities and risks of large language models in mental health. JMIR Mental Health, 11(1):e59479.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Daniel Ho Christopher Ré Adam Chilton Aditya K Alex Chohlas-Wood Austin Peters Brandon Wal- don Daniel Rockmore Diego Zambrano Dmitry Talisman Enam Hoque Faiz Surani Frank Fagan Galit Sarfaty Gregory Dickinson Haggai Porat Ja- son Hegland Jessica Wu Joe Nudell Joel Niklaus John Nay Jonathan Choi Kevin Tobia Margaret Ha- gan Megan Ma Michael Livermore Nikon Rasumov- Rahe Nils Holzenberger Noam Kolt Peter Hender- son Sean Rehaag Sharad Goel Shang Gao Spencer Williams Sunny Gandhi Tom Zur Varun Iyer Ze- hua Li Neel Guha, Julian Nyarko. 2023. Legalbench: A collaboratively built benchmark for measuring le- gal reasoning in large language models. Advances in Neural Information Processing Systems.
- Nick Obradovich, Sahib S Khalsa, Waqas U Khan, Jina Suh, Roy H Perlis, Olusola Ajilore, and Martin P Paulus. 2024. Opportunities and risks of large lan- guage models in psychiatry. NPP-Digital Psychia- try and Neuroscience, 2(1):8.
- Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vin- cent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830.
- Elizabeth C Stade, Shannon Wiltsey Stirman, Lyle H Ungar, Cody L Boland, H Andrew Schwartz, David B Yaden, João Sedoc, Robert J DeRubeis, Robb Willer, and Johannes C Eichstaedt. 2024. Large language models could change the future of behavioral health- care: a proposal for responsible development and evaluation. NPJ Mental Health Research, 3(1):12.
- Derald Wing Sue, David Sue, Helen A Neville, and Laura Smith. 2022. Counseling the culturally di- verse: Theory and practice. John Wiley & Sons.
- Alastair C Van Heerden, Julia R Pozuelo, and Bran- don A Kohrt. 2023. Global mental health services and the impact of artificial intelligence-powered large language models. JAMA psychiatry, 80(7):662- 664.
- Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Ev- geni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. 2020. Scipy 1.0: fundamental algorithms for scientific computing in python. Na- ture methods, 17(3):261-272.
- Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Mental- lama: interpretable mental health analysis on social media with large language models. In Proceedings of the ACM on Web Conference 2024, pages 4489- 4500.
- Phamornpun Yurayat and Supachai Tuklang. 2023. Uni- versity student counselees' attitudes and experiences towards online counseling during the covid-19 pan- demic: a mixed methods study. Journal of Higher Education Theory and Practice, 23(4).
- Masood Zangeneh and Alean Al-Krenawi. 2019. Cul- ture, diversity and mental health-enhancing clinical practice. Springer.
- Kai Zhang, Jun Yu, Zhiling Yan, Yixin Liu, Eashan Ad- hikarla, Sunyang Fu, Xun Chen, Chen Chen, Yuyin Zhou, Xiang Li, et al. 2023. Biomedgpt: a uni- fied and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv preprint arXiv:2305.17100. · arXiv
- Lexin Zhou, Wout Schellaert, Fernando Martínez- Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. 2024a. Larger and more in- structable language models become less reliable. Na- ture, pages 1-8.
Cited by
Sorted by citations
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204 · see citing works
- Authors
- Nguyen, V. C., Taher, M., Hong, D., Possobom, V. K., Gopalakrishnan, V. T., Raj, E., Li, Z., Soled, H. J., Birnbaum, M. L., Kumar, S., & De Choudhury, M.
- Affiliations
- Georgia Institute of Technology (US)
- Hofstra University (US)
- Columbia University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2410.22446
- OpenAlex
- W4404342259
- Cited by
- 3
- References
- 21
- Citations by year
- 2026: 1 · 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2230692)
- Georgia Institute of Technology
- National Institutes of Health
- National Institute of Mental Health (R01MH117172)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- large language models, mental health counseling, counselor competence, benchmarking, medical LLMs, fine-tuning, assessment and diagnosis, professional ethics, empathy, human oversight
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
