CoChat

When ELIZA Meets Therapists: A Turing Test for the Heart and Mind

Open access

The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.

Abstract

“Can machines be therapists?” is a question receiving increased attention given the relative ease of working with generative artificial intelligence. Although recent (and decades-old) research has found that humans struggle to tell the difference between responses from machines and humans1,2, recent findings suggest that artificial intelligence can write empathically and the generated content is rated highly by therapists and outperforms professionals.3,4,5,6,7,8 It is uncertain if in a preregistered competition where therapists and ChatGPT respond to therapeutic vignettes about couple therapy whether a) a panel of participants (N = 830) can tell which responses are ChatGPT-generated and which are written by therapists, b) the generated responses or the therapist-written responses fall more in line with key therapy principles, and c) linguistic differences between conditions are present. We show with a large sample (N = 830) that a) participants can rarely tell a difference between responses written by ChatGPT and responses written by a therapist, b) the responses written by ChatGPT are generally rated higher in key psychotherapy principles, and c) the language patterns between ChatGPT and therapists are different. Using different measures, we then confirm that responses written by ChatGPT were rated higher than the therapist's responses suggesting these differences may be explained by part-of-speech and response sentiment. This may be an early indication that ChatGPT has the potential to improve psychotherapeutic processes. We anticipate that this work may lead to the development of different methods of testing and creating psychotherapeutic interventions. Further, we discuss limitations (including the lack of the therapeutic context), and how continued research in this area may lead to improved efficacy of psychotherapeutic interventions allowing such interventions to be placed in the hands of individuals who need them the most.

Overview

Integrity signals

Registry

Crossref · 10.31219/osf.io/zyvr4

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 43 references

Cited by

See all 73

Metadata

Authors
Hatch, S. G., Goodman, Z. T., Vowels, L. M., Hatch, D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C., Harris, S. M., Holt, D. P., McLaughlin, M. M., O'Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S.
Affiliations
  • University of California San Diego (US)
  • University of Lausanne (CH)
  • Brigham Young University (US)
  • University of Miami (US)
  • University of Denver (US)
  • Utah Valley University (US)
  • Sam Houston State University (US)
  • University of Minnesota (US)
  • Massachusetts General Hospital (US)
Venue
PLOS mental health.
Published
2024-05-08
Volume
2
Issue
2
Pages
e0000145-e0000145
ISSN
2837-8156 (ISSN-L)
OpenAlex
W4407403557
Cited by
73
References
42
Citations by year
2026: 38 · 2025: 34 · 2024: 1
FWCI
57.59
Type
journal article
Language
en
Open access
Locations
  • PLOS mental health. (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
ChatGPT, Turing test, psychotherapy principles, AI-assisted psychotherapy, generative AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: When ELIZA Meets Therapists: A Turing Test for the Heart and Mind. https://app.cochat.ai/papers/10.31219/osf.io/zyvr4
The paper
Hatch, S. G., Goodman, Z. T., Vowels, L. M., Hatch, D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C., Harris, S. M., Holt, D. P., McLaughlin, M. M., O'Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S. (2024). When ELIZA Meets Therapists: A Turing Test for the Heart and Mind. PLOS mental health. https://doi.org/10.31219/osf.io/zyvr4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.