When ELIZA Meets Therapists: A Turing Test for the Heart and Mind
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
“Can machines be therapists?” is a question receiving increased attention given the relative ease of working with generative artificial intelligence. Although recent (and decades-old) research has found that humans struggle to tell the difference between responses from machines and humans1,2, recent findings suggest that artificial intelligence can write empathically and the generated content is rated highly by therapists and outperforms professionals.3,4,5,6,7,8 It is uncertain if in a preregistered competition where therapists and ChatGPT respond to therapeutic vignettes about couple therapy whether a) a panel of participants (N = 830) can tell which responses are ChatGPT-generated and which are written by therapists, b) the generated responses or the therapist-written responses fall more in line with key therapy principles, and c) linguistic differences between conditions are present. We show with a large sample (N = 830) that a) participants can rarely tell a difference between responses written by ChatGPT and responses written by a therapist, b) the responses written by ChatGPT are generally rated higher in key psychotherapy principles, and c) the language patterns between ChatGPT and therapists are different. Using different measures, we then confirm that responses written by ChatGPT were rated higher than the therapist's responses suggesting these differences may be explained by part-of-speech and response sentiment. This may be an early indication that ChatGPT has the potential to improve psychotherapeutic processes. We anticipate that this work may lead to the development of different methods of testing and creating psychotherapeutic interventions. Further, we discuss limitations (including the lack of the therapeutic context), and how continued research in this area may lead to improved efficacy of psychotherapeutic interventions allowing such interventions to be placed in the hands of individuals who need them the most.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Interestingly, contrary to popular belief, trained human therapists -in possession of human emotion and intuition -were not able to outperform the AI in terms of perceived empathy and helpfulness of their written relationship advice.”
From The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024 (2024) · cites this paper as [56]
In the citing paper
“Both, laypersons and therapists rated the textual AI-generated relationship advice as more empathetic and more helpful in comparison to the written advice provided by human therapists [48,56,58,59].”
From The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024 (2024) · cites this paper as [56]
Citations
Cites
brms : An R Package for Bayesian Multilevel Models Using Stan
Bürkner, P.-C. (2017) · Journal of Statistical Software · doi:10.18637/jss.v080.i01 · Cited by 9,476
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
NLTK: The Natural Language Toolkit
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,844
The Great Psychotherapy Debate: The Evidence for What Makes Psychotherapy Work
Wampold, B. E. (2015) · doi:10.4324/9780203582015 · Cited by 1,773
How important are the common factors in psychotherapy? An update
Wampold, B. E. (2015) · World Psychiatry · doi:10.1002/wps.20238 · Cited by 1,728
Ethical Principles of Psychologists and Code of Conduct.
(2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-030 · Cited by 1,679
Some implicit common factors in diverse methods of psychotherapy.
Rosenzweig, S. (1936) · American Journal of Orthopsychiatry · doi:10.1111/j.1939-0025.1936.tb05248.x · Cited by 1,042
Unified Protocol for Transdiagnostic Treatment of Emotional Disorders: A Randomized Controlled Trial
Farchione, T. J., Fairholme, C. P., Ellard, K. K., Boisseau, C. L., Thompson-Hollands, J., Carl, J. R., Gallagher, M. W., & Barlow, D. H. (2012) · Behavior Therapy · doi:10.1016/j.beth.2012.01.001 · Cited by 746
Cited by
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204
Hua, Y., Siddals, S., Ma, Z., Galatzer‐Levy, I., Xia, W., Hau, C., Na, H., Flathers, M., Linardon, J., Ayubcha, C., et al. (2025) · World Psychiatry · doi:10.1002/wps.21352 · Cited by 117
The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024
Döring, N., Le, T. D., Vowels, L. M., Vowels, M. J., & Marcantonio, T. L. (2024) · Current Sexual Health Reports · doi:10.1007/s11930-024-00397-y · Cited by 65
Self-citationPassmore, J., Olafsson, B., & Tee, D. (2025) · Journal of Work-Applied Management · doi:10.1108/jwam-11-2024-0164 · Cited by 79
Conversational AI in Pediatric Mental Health: A Narrative Review
Mansoor, M. A., Hamide, A., & Tran, T. (2025) · Children · doi:10.3390/children12030359 · Cited by 59
Metadata
- Authors
- Hatch, S. G., Goodman, Z. T., Vowels, L. M., Hatch, D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C., Harris, S. M., Holt, D. P., McLaughlin, M. M., O'Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S.
- Affiliations
- University of California San Diego (US)
- University of Lausanne (CH)
- Brigham Young University (US)
- University of Miami (US)
- University of Denver (US)
- Utah Valley University (US)
- Sam Houston State University (US)
- University of Minnesota (US)
- Massachusetts General Hospital (US)
- ORCID
- Venue
- PLOS mental health.
- Publisher
- Center for Open Science
- Published
- 2024-05-08
- Volume
- 2
- Issue
- 2
- Pages
- e0000145-e0000145
- ISSN
- 2837-8156 (ISSN-L)
- PMC
- PMC12798195
- OpenAlex
- W4407403557
- Cited by
- 73
- References
- 42
- Citations by year
- 2026: 38 · 2025: 34 · 2024: 1
- FWCI
- 57.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLOS mental health. (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- ChatGPT, Turing test, psychotherapy principles, AI-assisted psychotherapy, generative AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: When ELIZA Meets Therapists: A Turing Test for the Heart and Mind. https://app.cochat.ai/papers/10.31219/osf.io/zyvr4
- The paper
- Hatch, S. G., Goodman, Z. T., Vowels, L. M., Hatch, D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C., Harris, S. M., Holt, D. P., McLaughlin, M. M., O'Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S. (2024). When ELIZA Meets Therapists: A Turing Test for the Heart and Mind. PLOS mental health. https://doi.org/10.31219/osf.io/zyvr4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.