CoChat

When ELIZA meets therapists: A Turing test for the heart and mind

Open access

The DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index, and 1 are not checked yet; 3 of the 43 sources its claims cite were checked for support.

Abstract

“Can machines be therapists?” is a question receiving increased attention given the relative ease of working with generative artificial intelligence. Although recent (and decades-old) research has found that humans struggle to tell the difference between responses from machines and humans, recent findings suggest that artificial intelligence can write empathically and the generated content is rated highly by therapists and outperforms professionals. It is uncertain whether, in a preregistered competition where therapists and ChatGPT respond to therapeutic vignettes about couple therapy, a) a panel of participants can tell which responses are ChatGPT-generated and which are written by therapists (N = 13), b) the generated responses or the therapist-written responses fall more in line with key therapy principles, and c) linguistic differences between conditions are present. In a large sample (N = 830), we showed that a) participants could rarely tell the difference between responses written by ChatGPT and responses written by a therapist, b) the responses written by ChatGPT were generally rated higher in key psychotherapy principles, and c) the language patterns between ChatGPT and therapists were different. Using different measures, we then confirmed that responses written by ChatGPT were rated higher than the therapist’s responses suggesting these differences may be explained by part-of-speech and response sentiment. This may be an early indication that ChatGPT has the potential to improve psychotherapeutic processes. We anticipate that this work may lead to the development of different methods of testing and creating psychotherapeutic interventions. Further, we discuss limitations (including the lack of the therapeutic context), and how continued research in this area may lead to improved efficacy of psychotherapeutic interventions allowing such interventions to be placed in the hands of individuals who need them the most.

Verified 2026-10-07 · 1 reference not checked yet

Overview

CoChat Summary

TL;DR

In a preregistered Turing-test-style study, a US panel (N = 830) rarely distinguished ChatGPT 4.0 responses from therapist responses to 18 couple therapy vignettes, and ChatGPT responses were rated higher on common factors of therapy. ChatGPT responses also differed linguistically, with more positive sentiment and more nouns/adjectives even after controlling for length.

Key findings

  • Participants identified authors near chance: they correctly guessed therapist-authored responses 56.1% of the time and ChatGPT-authored responses 51.2% of the time, a 5.0 percentage-point difference (95% CI [3.2, 6.7]).
  • ChatGPT responses were rated higher on common factors than therapist responses (μ = 27.72 vs. 26.12; d = 1.63, 95% CI [1.49, 1.78]). Post hoc machine classification found ChatGPT responses more likely to be classified as connecting (d = 1.00, 95% CI [0.40, 1.59]), empathic (d = 0.75, 95% CI [0.11, 1.40]), and culturally competent (d = 0.81, 95% CI [0.17, 1.46]); effectiveness was not reliably different (d = 0.64, 95% CI [-0.03, 1.30]), and relevance was similar (d = -0.06, 95% CI [-0.77, 0.64]).
  • A post hoc attribution bias appeared: ChatGPT responses misattributed to therapists received the highest ratings (μ = 29.97), while therapist responses misattributed to ChatGPT received the lowest ratings (μ = 22.74); the author-by-attribution interaction was γ20 = -10.53, SE = 0.16.
  • Linguistically, ChatGPT responses had more positive sentiment (d = 0.92, 95% CI [0.32, 1.52]) and less negative sentiment (d = -1.04, 95% CI [-1.61, -0.47]); they were longer (IRR = 1.91, 95% CI [1.81, 2.02]) and had more nouns (IRR = 2.56, 95% CI [2.23, 2.96]), verbs (IRR = 2.56, 95% CI [2.23, 2.96]), adjectives (IRR = 2.78, 95% CI [2.23, 3.49]), adverbs (IRR = 1.64, 95% CI [1.31, 2.06]), and pronouns (IRR = 1.64, 95% CI [1.43, 1.88]). After controlling for length, ChatGPT still had more nouns (IRR = 1.31, 95% CI [1.10, 1.57]) and adjectives (IRR = 1.40, 95% CI [1.05, 1.88]), but similar verbs, adverbs, and pronouns.

Methods

  • Preregistered study using 18 couple therapy vignettes. Expert responses came from 13 experts (clinical psychology N = 9, counseling psychology N = 1, marriage and family therapy N = 2, psychiatry N = 1; 5–20+ years experience; most had Ph.D.s N = 9, Master’s N = 3, medical degree N = 1; 9 had couple therapy backgrounds). Experts were randomly assigned to two sets of 9 vignettes and peer-ranked the best responses. ChatGPT 4.0 responses were generated via an engineered prompt (no fine-tuning, no length limit, and ChatGPT was not shown therapist responses); the research team selected the best ChatGPT responses.
  • Panel participants (N = 830) were recruited via CloudResearch, representative of the US population; mean age was 45.17 years (SD = 16.56), 59.88% were in a current romantic relationship, and 18.07% had ever engaged in couple therapy. Participants were randomly assigned to receive a therapist-written or ChatGPT-generated response for each vignette, then guessed the author and rated common factors on five 7-point Likert items (CFA: CFI = .99, TLI = .99, RMSEA = .05, SRMR = .06; α = .94).
  • Bayesian analyses were used, with 95% credible intervals excluding zero as reliable; Cohen’s d for common factors and sentiment, incidence rate ratios (IRRs) for count outcomes, and brms regressions with Poisson family for counts. Sentiment and part-of-speech tagging used transforEmotion and UDPipe.

Limitations

  • Stated by the authors: limited number of vignettes; only one engineered prompt; small number of therapists; limited number of couple therapists; restrictive context; vignette responses may not generalize to actual therapy or to individual therapy; brief outcome measure; concerns about clinician sampling, response selection, and the therapist group; post hoc aims were not preregistered; small vignette sample (36 total responses, 18 therapist and 18 ChatGPT) led to imprecise estimates; findings need replication.
  • Evident from the text: the expert sample was small (N = 13) and included 4 experts without couple therapy backgrounds; the panel was general US adults rather than therapy clients or clinicians; the study used hypothetical vignettes rather than actual therapy sessions or confidential clinical data; and only one GenAI model (ChatGPT 4.0) was tested.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pmen.0000145

Cites

42 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 0 Not valid · 17 Not checked · 72 Own results and interpretations

39 more claims

Create a free account to see all 91 claims and to verify the 40 references still open.

See all claims — free account
  • Probably not a substantial claim. 29
  • It may state two separate claims as one. 24
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 2

  1. Hatch S. Leveraging Machine Learning to Optimize the Amount and Type of Practitioner Contact in Online Relationship Education for Low-Income Couples [Doctoral dissertation, University of Miami]. 2023. Available from: https://scholarship.miami.edu/esploro/outputs/doctoral/Leveraging-Machine-Learning-to-Optimize-the/991031799414502976.
  2. Bååth R. Bayesian first aid: A package that implements Bayesian alternatives to the classical*. test functions in R. In Proceedings of useR, 2014. 2014; p. 2.

Cited by

See all 77

Metadata

Authors
Hatch, S. G., Goodman, Z. T., Vowels, L., Hatch, H. D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C. R., Harris, S. M., Holt, D. P., McLaughlin, M., O’Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S. R.
Affiliations
  • University of California San Diego (US)
  • University of Lausanne (CH)
  • The Ohio State University (US)
  • Brigham Young University (US)
  • University of Miami (US)
  • University of Denver (US)
  • Utah Valley University (US)
  • Sam Houston State University (US)
  • University of Minnesota (US)
  • Massachusetts General Hospital (US)
  • Emory University (US)
Corresponding author
Hatch, H. Dorian
Venue
PLOS Mental Health
Published
2025-02-12
Volume
2
Issue
2
Pages
e0000145
ISSN
2837-8156 (ISSN-L)
OpenAlex
W4407403557
Cited by
73
References
42
Citations by year
2026: 38 · 2025: 34
FWCI
57.59
Type
journal article
Language
en
Open access
Locations
  • PLOS mental health. (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
ChatGPT, Turing test, psychotherapy principles, AI-assisted psychotherapy, generative AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.