When ELIZA Meets Therapists: A Turing Test for the Heart and Mind
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
“Can machines be therapists?” is a question receiving increased attention given the relative ease of working with generative artificial intelligence. Although recent (and decades-old) research has found that humans struggle to tell the difference between responses from machines and humans1,2, recent findings suggest that artificial intelligence can write empathically and the generated content is rated highly by therapists and outperforms professionals.3,4,5,6,7,8 It is uncertain if in a preregistered competition where therapists and ChatGPT respond to therapeutic vignettes about couple therapy whether a) a panel of participants (N = 830) can tell which responses are ChatGPT-generated and which are written by therapists, b) the generated responses or the therapist-written responses fall more in line with key therapy principles, and c) linguistic differences between conditions are present. We show with a large sample (N = 830) that a) participants can rarely tell a difference between responses written by ChatGPT and responses written by a therapist, b) the responses written by ChatGPT are generally rated higher in key psychotherapy principles, and c) the language patterns between ChatGPT and therapists are different. Using different measures, we then confirm that responses written by ChatGPT were rated higher than the therapist's responses suggesting these differences may be explained by part-of-speech and response sentiment. This may be an early indication that ChatGPT has the potential to improve psychotherapeutic processes. We anticipate that this work may lead to the development of different methods of testing and creating psychotherapeutic interventions. Further, we discuss limitations (including the lack of the therapeutic context), and how continued research in this area may lead to improved efficacy of psychotherapeutic interventions allowing such interventions to be placed in the hands of individuals who need them the most.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Interestingly, contrary to popular belief, trained human therapists -in possession of human emotion and intuition -were not able to outperform the AI in terms of perceived empathy and helpfulness of their written relationship advice.”
From The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024 (2024) · cites this paper as [56]
In the citing paper
“Both, laypersons and therapists rated the textual AI-generated relationship advice as more empathetic and more helpful in comparison to the written advice provided by human therapists [48,56,58,59].”
From The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024 (2024) · cites this paper as [56]
Citations
Cites
brms : An R Package for Bayesian Multilevel Models Using Stan
Bürkner, P.-C. (2017) · Journal of Statistical Software · doi:10.18637/jss.v080.i01 · Cited by 9,476
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
NLTK: The Natural Language Toolkit
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,844
The Great Psychotherapy Debate: The Evidence for What Makes Psychotherapy Work
Wampold, B. E. (2015) · doi:10.4324/9780203582015 · Cited by 1,773
How important are the common factors in psychotherapy? An update
Wampold, B. E. (2015) · World Psychiatry · doi:10.1002/wps.20238 · Cited by 1,728
Ethical Principles of Psychologists and Code of Conduct.
(2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-030 · Cited by 1,679
Some implicit common factors in diverse methods of psychotherapy.
Rosenzweig, S. (1936) · American Journal of Orthopsychiatry · doi:10.1111/j.1939-0025.1936.tb05248.x · Cited by 1,042
Unified Protocol for Transdiagnostic Treatment of Emotional Disorders: A Randomized Controlled Trial
Farchione, T. J., Fairholme, C. P., Ellard, K. K., Boisseau, C. L., Thompson-Hollands, J., Carl, J. R., Gallagher, M. W., & Barlow, D. H. (2012) · Behavior Therapy · doi:10.1016/j.beth.2012.01.001 · Cited by 746
Cited by
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204
Hua, Y., Siddals, S., Ma, Z., Galatzer‐Levy, I., Xia, W., Hau, C., Na, H., Flathers, M., Linardon, J., Ayubcha, C., et al. (2025) · World Psychiatry · doi:10.1002/wps.21352 · Cited by 117
The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024
Döring, N., Le, T. D., Vowels, L. M., Vowels, M. J., & Marcantonio, T. L. (2024) · Current Sexual Health Reports · doi:10.1007/s11930-024-00397-y · Cited by 65
Self-citationPassmore, J., Olafsson, B., & Tee, D. (2025) · Journal of Work-Applied Management · doi:10.1108/jwam-11-2024-0164 · Cited by 79
Conversational AI in Pediatric Mental Health: A Narrative Review
Mansoor, M. A., Hamide, A., & Tran, T. (2025) · Children · doi:10.3390/children12030359 · Cited by 59
Trusting emotional support from generative artificial intelligence: a conceptual review
Volpato, R., DeBruine, L. M., & Stumpf, S. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100195 · Cited by 72
Emerging AI individualism: how young people integrate social AI into everyday life
Brandtzaeg, P. B., Følstad, A., & Skjuve, M. (2025) · Communication and Change · doi:10.1007/s44382-025-00011-2 · Cited by 30
Blease, C., Kharko, A., Sanchez, C. G., Alderman, J. E., Kumpunen, S., Sundemo, D. S., & Torous, J. (2025) · Journal of Medical Internet Research · doi:10.2196/74428 · Cited by 36
Transforming mental health research and care through artificial intelligence
Opel, N., & Breakspear, M. J. (2026) · Science · doi:10.1126/science.adz9193 · Cited by 37
Artificial intelligence and psychoanalysis: is it time for psychoanalyst.AI?
Rabeyron, T. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1558513 · Cited by 25
Therapy chatbots and emotional complexity: do therapy chatbots really empathise?
Gabriels, K., & Goffin, K. (2026) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2025.102263 · Cited by 19
Characterizing Delusional Spirals through Human-LLM Chat Logs
Moore, J., Mehta, A., Agnew, W., Anthis, J. R., Louie, R., Mai, Y., Yin, P., Cheng, M., Paech, S. J., Klyman, K., et al. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3806443 · Cited by 15
Human vs. Machine: Comparing AI-Generated and Human-Written Psychological Reports
Lockwood, A. B., Farmer, R. L., Shergill, G., Benson, N. F., & Gilbert, K. (2025) · Journal of Psychoeducational Assessment · doi:10.1177/07342829251346623 · Cited by 16
Blease, C. (2025) · JMIR Mental Health · doi:10.2196/78663 · Cited by 11
Faria, V., Goturi, N., Dynak, A., Talbert, C., Pondelis, N. J., Annoni, M., Blease, C., Holmes, S. A., & Moulton, E. A. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1612215 · Cited by 12
Frisone, F., Pupillo, C., Rossi, C., & Riva, G. (2025) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2025.45510.cyeuro · Cited by 9
Vecchione, B., & Singh, R. (2025) · Big Data & Society · doi:10.1177/20539517251383884 · Cited by 12
Therapy as an NLP Task: Comparing LLMs and Human Peers Behaviors in CBT Sessions
Iftikhar, Z., Ransom, S., Xiao, A. W., Nugent, N. R., & Huang, J. J. S. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3817112 · Cited by 42
Vowels, L. M., Sweeney, S. K., & Vowels, M. J. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100141 · Cited by 14
Self-citationDesage, C., Patterson, T., Wu, Y., Bechtel, A., Bharat, A., Vaclavik, D., & Bunge, E. (2025) · Psychiatry Research · doi:10.1016/j.psychres.2025.116900 · Cited by 3
Navigating relationships with GenAI chatbots: User attitudes, acceptability, and potential
Vowels, L. M., Francois‐Walcott, R. R. R., Grandjean, M., Darwiche, J., & Vowels, M. J. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100183 · Cited by 7
Self-citationCorrection: When ELIZA meets therapists: A Turing test for the heart and mind
Hatch, S. G., Goodman, Z. T., Vowels, L. M., Hatch, H. D., Brown, A. L., Guttman, S., Le, Y., Bailey, B. M., Bailey, R. J., Esplin, C. R., et al. (2025) · PLOS mental health. · doi:10.1371/journal.pmen.0000426 · Cited by 5
Self-citationAnti-AI Bias Toward Couple Images and Couple Counseling: Findings from Two Experiments
Döring, N. M., & Mohseni, M. R. (2025) · Archives of Sexual Behavior · doi:10.1007/s10508-025-03318-9 · Cited by 9
AI aversion? Effects of author disclosure on young people’s perceptions of mental health advice
Brandtzæg, P. B., Skjuve, M. B., & Følstad, A. (2026) · Cyberpsychology Journal of Psychosocial Research on Cyberspace · doi:10.5817/cp2026-2-1 · Cited by 4
Are Psychiatry Faculty Prepared to Guide AI Use? A National Survey
Mercado, A. E., Ferguson, L., Chastain, J., Williams, J., Nahhas, R. W., Tripathi, S., & Grant, L. (2026) · Academic Psychiatry · doi:10.1007/s40596-026-02352-6 · Cited by 4
Ladegaard, I. (2025) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231251326981 · Cited by 4
Frisone, F., Pupillo, C., Rossi, C., & Riva, G. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1603238 · Cited by 4
Döring, N. M., Mikhailova, V., & Mohseni, M. R. (2026) · Archives of Sexual Behavior · doi:10.1007/s10508-025-03382-1 · Cited by 4
Clinical decision-making and artificial intelligence: The role of large language models in medicine
El-Sayid, W., Salami, K., Bhasin, R., Abdul-Nabi, M., Elashmawy, A. E., & Jaruzel, M. E. (2025) · Clinical Research in Practice The Journal of Team Hippocrates · doi:10.22237/crp/1743681960 · Cited by 1
Responsible Artificial Intelligence in Precision Psychiatry
Vesterinen, T. (2025) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-031-99654-2_8 · Cited by 1
AI and the coming mental health zombie apocalypse
Kaplan, D. M., Palitsky, R., & Raison, C. L. (2025) · Molecular Psychiatry · doi:10.1038/s41380-025-03323-3 · Cited by 1
Merikko, J., & Räsänen, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-13184-3_14 · Cited by 3
One-Sided Parasociality with AI
Degen, J. L. (2025) · Palgrave Studies in Cyberpsychology. · doi:10.1007/978-3-032-07438-6_5 · Cited by 2
Федоровна, Ф. И., & Shlyapov, I. V. (2025) · Russian Journal of Education and Psychology · doi:10.12731/2658-4034-2025-16-5-829 · Cited by 3
Crocamo, C. (2026) · BJPsych Advances · doi:10.1192/bja.2025.10191 · Cited by 1
Scaling Empathy: How AI Enhances Advisor Communication Skills to Promote Financial Wellness
Lynn, C. (2026) · Financial Planning Review · doi:10.1002/cfp2.70025 · Cited by 1
Varianten und Risikoaspekte des psychologisch-therapeutischen KI-Einsatzes
Wanderwitz, M., & Weber, E. (2026) · Springer eBooks · doi:10.1007/978-3-662-72661-7_12 · Cited by 1
Tulbure, B. T., Spătaru, B., Rusu, A., Faur, D., Precupaș, D., Trancă, L. M., Popa, S. N., & Spătaru, A. (2026) · JMIR Formative Research · doi:10.2196/84296 · Cited by 2
Vowels, L. M., Vohra, P., Li, D., Zeinoddin, P. S., Elswick, A., Marcantonio, T. L., Wood, N. D., & Vowels, M. J. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001352 · Cited by 2
Self-citationArtificial Empathy: Evaluating AI Emotional Support Systems in Organizations
Ahmad, F., & Phillips, N. (2026) · Springer eBooks · doi:10.1007/978-3-032-12670-2_13 · Cited by 2
Herbener, A. B., Zachariae, R. H., Klincewicz, M., Thøgersen, M. B., Hermann, M. R., & Damholdt, M. F. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100347 · Cited by 1
李, 柠. (2025) · Advances in Education · doi:10.12677/ae.2025.155736 · Cited by 1
Generative AI in Primary Care: A qualitative study of UK General Practitioners’ Views (Preprint)Preprint
Blease, C., Sanchez, C. G., Locher, C., McMillan, B., Gaab, J., & Torous, J. B. (2025) · doi:10.2196/preprints.74428
Effectiveness of AI Psychotherapy, Compared to Human Providers: A Meta-Synthesis Research
Hidalgo, R. G. (2025) · doi:10.46569/9w032c28t · Cited by 1
Sprachmodelle in der Diagnostik und Therapie psychischer Erkrankungen
Kambeitz, J. P. (2025) · DNP – Die Neurologie & Psychiatrie · doi:10.1007/s15202-025-6543-5
Li, Y., & Liu, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-7437820/v1 · Cited by 1
Lack, C. W., & Haws, J. K. (2025) · Psychiatric Clinics of North America · doi:10.1016/j.psc.2025.08.018
Nardin, K. D. L., Silva, D. C. A., Costa, E. R., Justi, M. M., & Oliveira, F. C. S. D. (2025) · Psicologia e Saúde em Debate · doi:10.22289/2446-922x.v11a2a26
Artificial Intelligence and Student Mental Health: Risks and Opportunities
Akat, M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-6375-2.ch002 · Cited by 2
Possible risks of turning to tea-bots based on artificial intelligence in difficult life situations
Enikolopov, S. N., Boyko, O. M., Медведева, Т. Е., & Воронцова, О. Ю. (2025) · Медицинская психология в России · doi:10.24884/2219-8245-2025-17-4-69-82 · Cited by 1
Kuhlmeier, F. O., Hanschmann, L., Rabe, M., Lüttke, S., Brakemeier, E., & Maedche, A. (2026) · JMIR Mental Health · doi:10.2196/94781 · Cited by 1
Künstliche Intelligenz in der Psychoonkologie – Anwendungsmöglichkeiten und Implikationen
Sauer, C., Maatouk, I., & Haussmann, A. (2026) · Die Onkologie · doi:10.1007/s00761-026-01939-w
AI Helps Those Who Have Less? Social Support Gaps and Perceptions of AI Emotional Support
Chen, W., Hou, Y. T.-Y., Chen, R. Y.-S., Tai, C.-H., Weng, C. C.-H., & Chen, Y. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798800
Shirvani, M. S., Crowley, J., Peng, C., Liu, J., Chao, T. W.-C., Martinez, S., Brandt, L., Kim, I.-J., & Yoon, D. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790986 · Cited by 4
Banfi, G., & Crescentini, N. (2026) · Health, Risk & Society · doi:10.1080/13698575.2026.2663062
Liljedahl, N., Ali, L., Liljedahl, S. I., Falk, Ö., & Steingrímsson, S. (2026) · doi:10.2196/preprints.98840
Deceptive Empathy in Large Language Model Responses to Suicide-Related Disclosures (Preprint)Preprint
Keskin, Y., Park, T., & Bozkurt, S. (2026) · doi:10.2196/preprints.99710
Richter, M. (2026) · Springer eBooks · doi:10.1007/978-3-032-16800-9_10
Generated Hover-to-Reveal Buttons Gate Exposure at the Point of Choice in a Goal-Reflection Chat
Göldi, A., Rietsche, R., & Ungar, L. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2664697
Deceptive Empathy in Large Language Model Responses to Suicide-Related Disclosures (Preprint)Preprint
Keskin, Y., Park, T., & Bozkurt, S. (2026) · doi:10.2196/preprints.105273
Walther, M., Taubner, S., & Gertz, M. (2026) · Die Psychotherapie · doi:10.1007/s00278-026-00866-w
Can AI Heal? A Conceptual Exploration of the Potential of AI in Mental Healthcare
Nair, H. B., & Vowels, L. M. (2026) · Springer eBooks · doi:10.1007/978-3-032-23626-5_2
Self-citationSchnittstelle Technologie und Wissenschaft
Mundorf, A., Peterburs, J., & Ocklenburg, S. (2026) · Springer eBooks · doi:10.1007/978-3-662-73259-5_14
Łabuz, M., Szczęsny, P., & Mika-Łabuz, K. (2026) · Ethics and Information Technology · doi:10.1007/s10676-026-09917-x · Cited by 3
Liljedahl, N., Ali, L., Liljedahl, S., Falk, Ö., & Steingrímsson, S. (2026) · JMIR Mental Health · doi:10.2196/98840
Internet-/Online-basierte Psychotherapieverfahren
Zagorscak, P., & Knaevelsrud, C. (2026) · Springer Reference Medizin · doi:10.1007/978-3-642-45028-0_130-1
Yang, H., Chang, F., Liu, Z., Zhang, W., & Cai, J. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103535
Cornelius‐White, J. H. D., & Kanamori, Y. (2026) · The Journal of Humanistic Counseling · doi:10.1002/johc.70062
Ethische Aspekte zur KI-Anwendung in der Psychotherapie
Schürmann, J. M., Sedláková, J., & Trachsel, M. (2026) · Psychotherapie-Wissenschaft · doi:10.30820/1664-9583-2026-2-17
Subjectivités et environnements numériques
Rabeyron, T., & Potier, R. (2026) · Dunod eBooks · doi:10.3917/dunod.jung.2026.01.0236
Hlynsson, J. I., Andersson, G., & Carlbring, P. (2026) · Internet Interventions · doi:10.1016/j.invent.2026.101012
Woerner, J. I., Marcantonio, T. L., & Castillo, H. L. (2026) · Archives of Sexual Behavior · doi:10.1007/s10508-026-03594-z
Metadata
- Authors
- Hatch, S. G., Goodman, Z. T., Vowels, L. M., Hatch, D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C., Harris, S. M., Holt, D. P., McLaughlin, M. M., O'Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S.
- Affiliations
- University of California San Diego (US)
- University of Lausanne (CH)
- Brigham Young University (US)
- University of Miami (US)
- University of Denver (US)
- Utah Valley University (US)
- Sam Houston State University (US)
- University of Minnesota (US)
- Massachusetts General Hospital (US)
- ORCID
- Venue
- PLOS mental health.
- Publisher
- Center for Open Science
- Published
- 2024-05-08
- Volume
- 2
- Issue
- 2
- Pages
- e0000145-e0000145
- ISSN
- 2837-8156 (ISSN-L)
- PMC
- PMC12798195
- OpenAlex
- W4407403557
- Cited by
- 73
- References
- 42
- Citations by year
- 2026: 38 · 2025: 34 · 2024: 1
- FWCI
- 57.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLOS mental health. (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- ChatGPT, Turing test, psychotherapy principles, AI-assisted psychotherapy, generative AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.