When ELIZA meets therapists: A Turing test for the heart and mind
Open accessThe DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index, and 1 are not checked yet; 3 of the 43 sources its claims cite were checked for support.
Abstract
“Can machines be therapists?” is a question receiving increased attention given the relative ease of working with generative artificial intelligence. Although recent (and decades-old) research has found that humans struggle to tell the difference between responses from machines and humans, recent findings suggest that artificial intelligence can write empathically and the generated content is rated highly by therapists and outperforms professionals. It is uncertain whether, in a preregistered competition where therapists and ChatGPT respond to therapeutic vignettes about couple therapy, a) a panel of participants can tell which responses are ChatGPT-generated and which are written by therapists (N = 13), b) the generated responses or the therapist-written responses fall more in line with key therapy principles, and c) linguistic differences between conditions are present. In a large sample (N = 830), we showed that a) participants could rarely tell the difference between responses written by ChatGPT and responses written by a therapist, b) the responses written by ChatGPT were generally rated higher in key psychotherapy principles, and c) the language patterns between ChatGPT and therapists were different. Using different measures, we then confirmed that responses written by ChatGPT were rated higher than the therapist’s responses suggesting these differences may be explained by part-of-speech and response sentiment. This may be an early indication that ChatGPT has the potential to improve psychotherapeutic processes. We anticipate that this work may lead to the development of different methods of testing and creating psychotherapeutic interventions. Further, we discuss limitations (including the lack of the therapeutic context), and how continued research in this area may lead to improved efficacy of psychotherapeutic interventions allowing such interventions to be placed in the hands of individuals who need them the most.
Verified 2026-10-07 · 1 reference not checked yet
Overview
CoChat Summary
TL;DR
In a preregistered Turing-test-style study, a US panel (N = 830) rarely distinguished ChatGPT 4.0 responses from therapist responses to 18 couple therapy vignettes, and ChatGPT responses were rated higher on common factors of therapy. ChatGPT responses also differed linguistically, with more positive sentiment and more nouns/adjectives even after controlling for length.
Key findings
- Participants identified authors near chance: they correctly guessed therapist-authored responses 56.1% of the time and ChatGPT-authored responses 51.2% of the time, a 5.0 percentage-point difference (95% CI [3.2, 6.7]).
- ChatGPT responses were rated higher on common factors than therapist responses (μ = 27.72 vs. 26.12; d = 1.63, 95% CI [1.49, 1.78]). Post hoc machine classification found ChatGPT responses more likely to be classified as connecting (d = 1.00, 95% CI [0.40, 1.59]), empathic (d = 0.75, 95% CI [0.11, 1.40]), and culturally competent (d = 0.81, 95% CI [0.17, 1.46]); effectiveness was not reliably different (d = 0.64, 95% CI [-0.03, 1.30]), and relevance was similar (d = -0.06, 95% CI [-0.77, 0.64]).
- A post hoc attribution bias appeared: ChatGPT responses misattributed to therapists received the highest ratings (μ = 29.97), while therapist responses misattributed to ChatGPT received the lowest ratings (μ = 22.74); the author-by-attribution interaction was γ20 = -10.53, SE = 0.16.
- Linguistically, ChatGPT responses had more positive sentiment (d = 0.92, 95% CI [0.32, 1.52]) and less negative sentiment (d = -1.04, 95% CI [-1.61, -0.47]); they were longer (IRR = 1.91, 95% CI [1.81, 2.02]) and had more nouns (IRR = 2.56, 95% CI [2.23, 2.96]), verbs (IRR = 2.56, 95% CI [2.23, 2.96]), adjectives (IRR = 2.78, 95% CI [2.23, 3.49]), adverbs (IRR = 1.64, 95% CI [1.31, 2.06]), and pronouns (IRR = 1.64, 95% CI [1.43, 1.88]). After controlling for length, ChatGPT still had more nouns (IRR = 1.31, 95% CI [1.10, 1.57]) and adjectives (IRR = 1.40, 95% CI [1.05, 1.88]), but similar verbs, adverbs, and pronouns.
Methods
- Preregistered study using 18 couple therapy vignettes. Expert responses came from 13 experts (clinical psychology N = 9, counseling psychology N = 1, marriage and family therapy N = 2, psychiatry N = 1; 5–20+ years experience; most had Ph.D.s N = 9, Master’s N = 3, medical degree N = 1; 9 had couple therapy backgrounds). Experts were randomly assigned to two sets of 9 vignettes and peer-ranked the best responses. ChatGPT 4.0 responses were generated via an engineered prompt (no fine-tuning, no length limit, and ChatGPT was not shown therapist responses); the research team selected the best ChatGPT responses.
- Panel participants (N = 830) were recruited via CloudResearch, representative of the US population; mean age was 45.17 years (SD = 16.56), 59.88% were in a current romantic relationship, and 18.07% had ever engaged in couple therapy. Participants were randomly assigned to receive a therapist-written or ChatGPT-generated response for each vignette, then guessed the author and rated common factors on five 7-point Likert items (CFA: CFI = .99, TLI = .99, RMSEA = .05, SRMR = .06; α = .94).
- Bayesian analyses were used, with 95% credible intervals excluding zero as reliable; Cohen’s d for common factors and sentiment, incidence rate ratios (IRRs) for count outcomes, and brms regressions with Poisson family for counts. Sentiment and part-of-speech tagging used transforEmotion and UDPipe.
Limitations
- Stated by the authors: limited number of vignettes; only one engineered prompt; small number of therapists; limited number of couple therapists; restrictive context; vignette responses may not generalize to actual therapy or to individual therapy; brief outcome measure; concerns about clinician sampling, response selection, and the therapist group; post hoc aims were not preregistered; small vignette sample (36 total responses, 18 therapist and 18 ChatGPT) led to imprecise estimates; findings need replication.
- Evident from the text: the expert sample was small (N = 13) and included 4 experts without couple therapy backgrounds; the panel was general US adults rather than therapy clients or clinicians; the study used hypothetical vignettes rather than actual therapy sessions or confidential clinical data; and only one GenAI model (ChatGPT 4.0) was tested.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 0 Not valid · 17 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“existing work only focuses on GenAI’s capacity to produce responses high in perceived empathy, one of the five key elements of the therapeutic process according to the common factor metatheory.”
§ When ELIZA meets therapists: A Turing test for the heart and mind > When AI meets couple therapy: A Turing test for the heart and mind > Limitations in the current literature
What the sources say
- Not checked yetcites [27] K Laska, 2014, Expanding the lens of evidence-based practice in psychotherapy: A common factors perspective, Psychother source
- Partly supportedcites [28] B. Wampold, 2015, How important are the common factors in psychotherapy? An update, World Psychiatry sourceExperimental — accuracy not yet measured
Our copy supports the core attribution: the 'Empathy and related constructs' section states that 'Therapist expressed empathy is a primary common factor, critical to pathway 1 of the contextual model,' and the Abstract lists empathy among the common factors presented as 'important therapeutic elements.' However, the paper never enumerates 'five key elements of the therapeutic process': 'THE CONTEXTUAL MODEL' specifies three pathways (real relationship, creation of expectations, enactment of health promoting actions) preceded by an initial therapeutic relationship, and the abstract's five-item list (alliance, empathy, expectations, cultural adaptation, therapist differences) is a list of factors reviewed, not of constitutive elements. The source also calls it a common factor 'model' (the contextual model), noting other common factor models exist, rather than a 'metatheory,' so that framing and the count of five are not attributable to Wampold (2015). (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“transcripts of sessions from randomized controlled trials are rarely available [13, 25, 26].”
§ When ELIZA meets therapists: A Turing test for the heart and mind > When AI meets couple therapy: A Turing test for the heart and mind > Limitations in the current literature
What the sources say
- Not checked yetcites [13] B Doss, 2020, Online programs improve relationship functioning for distressed low-income couples: Results from a nationwide randomized controlled trial, J Consult Clin Psychol source
- Could not verifycites [25] T Farchione, 2012, Unified protocol for transdiagnostic treatment of emotional disorders: a randomized controlled trial, Behav Ther sourceExperimental — accuracy not yet measured
Our copy of Farchione et al. (2012) contains no statement about the availability of session transcripts from randomized controlled trials. The closest passages concern this trial's own methods: the Method section 'THERAPISTS AND TREATMENT INTEGRITY' notes adherence 'was closely monitored during weekly supervision and manual development meetings, though it was not systematically assessed,' and the Discussion limitations repeat that 'nor was treatment fidelity evaluated in this trial' — neither is a claim about transcripts being rarely available across RCTs. Searches for 'transcripts of sessions', 'session recordings / audiotapes / data sharing', and 'treatment fidelity adherence' returned only these passages plus unrelated Results/References text. (LLM, conf=0.70)
- Not checked yetcites [26] A Christensen, 2004, Traditional versus integrative behavioral couple therapy for significantly and chronically distressed married couples., J Consult Clin Psychol source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although this difference was statistically reliable, accurate identification within groups was only marginally better than chance and accurate identification in the between group comparison was close to zero (i.e., 5%; see Table 2).”
§ When ELIZA meets therapists: A Turing test for the heart and mind > Results > Aim 1: Can a panel of participants tell the difference between responses written by a knowing expert and responses created using GenAI?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the responses written by ChatGPT were generally rated higher in key psychotherapy principles”
§ When ELIZA meets therapists: A Turing test for the heart and mind > Abstract
plainn = 830- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the language patterns between ChatGPT and therapists were different.”
§ When ELIZA meets therapists: A Turing test for the heart and mind > Abstract
plainn = 830- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
39 more claims
Create a free account to see all 91 claims and to verify the 40 references still open.
- Probably not a substantial claim. 29
- It may state two separate claims as one. 24
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This architecture enables LLMs to maintain coherence within complex, multi‐turn dialogues in ways that previous approaches could not achieve”
From Charting the evolution of artificial intelligence mental health chatbots from rule‐based systems to large language models: a systematic review (2025) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ When ELIZA meets therapists: A Turing test for the heart and mind > Method > Ethics and data availability statement
“Before panel data were collected, all analyses, hypotheses, and tests were preregistered”
§ When ELIZA meets therapists: A Turing test for the heart and mind > Discussion
“Third, post hoc aims were not preregistered and were explorations that took place after examining the data.”
Citations
Cites
brms : An R Package for Bayesian Multilevel Models Using Stan
Bürkner, P.-C. (2017) · Journal of Statistical Software · doi:10.18637/jss.v080.i01 · Cited by 9,476
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
NLTK: The Natural Language Toolkit
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,844
The Great Psychotherapy Debate: The Evidence for What Makes Psychotherapy Work
Wampold, B. E. (2015) · doi:10.4324/9780203582015 · Cited by 1,773
How important are the common factors in psychotherapy? An update
Wampold, B. E. (2015) · World Psychiatry · doi:10.1002/wps.20238 · Cited by 1,728
Ethical Principles of Psychologists and Code of Conduct.
(2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-030 · Cited by 1,679 · cited 1×
“However, supervision is required during this training process to ensure participant safety [23].”
Some implicit common factors in diverse methods of psychotherapy.
Rosenzweig, S. (1936) · American Journal of Orthopsychiatry · doi:10.1111/j.1939-0025.1936.tb05248.x · Cited by 1,042
Unified Protocol for Transdiagnostic Treatment of Emotional Disorders: A Randomized Controlled Trial
Farchione, T. J., Fairholme, C. P., Ellard, K. K., Boisseau, C. L., Thompson-Hollands, J., Carl, J. R., Gallagher, M. W., & Barlow, D. H. (2012) · Behavior Therapy · doi:10.1016/j.beth.2012.01.001 · Cited by 746
Research works printed but not linked: 3
- S. Hatch, 2023, Leveraging Machine Learning to Optimize the Amount and Type of Practitioner Contact in Online Relationship Education for Low-Income Couples [Doctoral dissertation, University of Miami].
- A Tomašević, 2024, Decoding Emotion Dynamics in Videos using Dynamic Exploratory Graph Analysis and Zero-Shot Image Classification: A Simulation and Tutorial using the transforEmotion R package., PsyArXiv Preprint
- R. Bååth, 2014, Bayesian first aid: A package that implements Bayesian alternatives to the classical*. test functions in R., In Proceedings of useR, 2
Cited by
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204
Hua, Y., Siddals, S., Ma, Z., Galatzer‐Levy, I., Xia, W., Hau, C., Na, H., Flathers, M., Linardon, J., Ayubcha, C., et al. (2025) · World Psychiatry · doi:10.1002/wps.21352 · Cited by 117
“This architecture enables LLMs to maintain coherence within complex, multi-turn dialogues in ways that previous approaches could not achieve 21 .”
Large language models as mental health resources: Patterns of use in the United States.
Rousmaniere, T., Zhang, Y., Li, X., & Shah, S. (2025) · Practice Innovations · doi:10.1037/pri0000292 · Cited by 71
The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024
Döring, N., Le, T. D., Vowels, L. M., Vowels, M. J., & Marcantonio, T. L. (2024) · Current Sexual Health Reports · doi:10.1007/s11930-024-00397-y · Cited by 65
“Both, layper-sons and therapists rated the textual AI-generated relationship advice as more empathetic and more helpful in comparison to the written advice provided by human therapists [48, 56, 58, 59].”
Self-citationPassmore, J., Olafsson, B., & Tee, D. (2025) · Journal of Work-Applied Management · doi:10.1108/jwam-11-2024-0164 · Cited by 79
Metadata
- Authors
- Hatch, S. G., Goodman, Z. T., Vowels, L., Hatch, H. D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C. R., Harris, S. M., Holt, D. P., McLaughlin, M., O’Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S. R.
- Affiliations
- University of California San Diego (US)
- University of Lausanne (CH)
- The Ohio State University (US)
- Brigham Young University (US)
- University of Miami (US)
- University of Denver (US)
- Utah Valley University (US)
- Sam Houston State University (US)
- University of Minnesota (US)
- Massachusetts General Hospital (US)
- Emory University (US)
- Corresponding author
- Hatch, H. Dorian
- ORCID
- Venue
- PLOS Mental Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2025-02-12
- Volume
- 2
- Issue
- 2
- Pages
- e0000145
- ISSN
- 2837-8156 (ISSN-L)
- PMC
- PMC12798195
- OpenAlex
- W4407403557
- Semantic Scholar
- fa537752421cb772ad2e9c7b934584b810cfaabf
- Cited by
- 73
- References
- 42
- Citations by year
- 2026: 38 · 2025: 34
- FWCI
- 57.59
- Type
- journal article
- Language
- en
- Update notices
- correction (2025-08-29)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS mental health. (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- ChatGPT, Turing test, psychotherapy principles, AI-assisted psychotherapy, generative AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: When ELIZA meets therapists: A Turing test for the heart and mind. https://app.cochat.ai/papers/10.1371/journal.pmen.0000145
- The paper
- Hatch, S. G., Goodman, Z. T., Vowels, L., Hatch, H. D., Brown, A. L., Guttman, S., Le, Y., Bailey, B., Bailey, R. J., Esplin, C. R., Harris, S. M., Holt, D. P., McLaughlin, M., O’Connell, P., Rothman, K., Ritchie, L., Top, D. N., & Braithwaite, S. R. (2025). When ELIZA meets therapists: A Turing test for the heart and mind. PLOS Mental Health. https://doi.org/10.1371/journal.pmen.0000145
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.