Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index; 18 of the 29 sources its claims cite were checked for support.

People cannot distinguish GPT-4 from a human in a Turing test

Open access
Jones, Cameron Robert, Rathi, Ishika, Taylor, Sydney, Bergen, Benjamin K.·2025·Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency

Abstract

AI systems that can fool people into thinking that they are human could have widespread social and economic consequences.In order to measure this ability, we evaluated 3 systems (ELIZA, GPT-3.5 and GPT-4) in a randomized, controlled, and preregistered Turing test.Human participants had a 5 minute conversation with either a human or an AI, and judged whether or not they thought their interlocutor was human.GPT-4 was judged to be a human 54% of the time, significantly more often than ELIZA (22%) but less often than actual humans (67%).In order to test the generalizability of our results, we replicated the study on a second population (undergraduate students) and found that the same prompt with GPT-4o achieved a pass rate of 77%, slightly higher than the human pass rate of 71%.On some interpretations, the results provide the first robust empirical demonstration that any artificial system passes an interactive 2-player Turing test.The results have implications for debates around machine intelligence and, more urgently, suggest that deception by current AI systems may go undetected.Analysis of participants' strategies and reasoning suggests that stylistic and socio-emotional factors play a larger role in passing the Turing test than traditional notions of intelligence.We release the full transcripts of the replication data to enable further investigation of human-AI interaction dynamics and deception.

Overview

CoChat Summary

TL;DR

In a preregistered, randomized two-party Turing test, human interrogators judged GPT-4 to be human 54% of the time—statistically indistinguishable from chance—while judging actual humans human 67% of the time and ELIZA only 22%. A conceptual replication with GPT-4o on an undergraduate sample found a 76% pass rate, slightly above the 71% human pass rate, suggesting stylistic and socio-emotional cues matter more than traditional notions of intelligence.

Key findings

  • Main study pass rates: GPT-4 54%, GPT-3.5 50%, ELIZA 22%, humans 67%. GPT-4's pass rate was not significantly different from chance (z = 0.799, p = 0.424), while ELIZA's was significantly below chance (z = -5.243, p < 0.001), serving as a manipulation check.
  • GPT-4's pass rate was significantly higher than ELIZA's (z = 4.11, p < 0.001) and significantly lower than humans' (z = 2.42, p = 0.017), but did not differ from GPT-3.5 (z = 0.497, p = 0.619). Bayesian estimates put GPT-4's pass rate between 44% and 63% and ELIZA's between 15% and 31%.
  • Replication (GPT-4o, tiktok prompt): 76% pass rate across 63 games, significantly above chance (z = 3.93, p < 0.001) and above ELIZA (z = 5.47, p < 0.001), but not significantly different from human witnesses at 71% (z = -0.735, p = 0.462). A minimal prompt achieved only 25% (z = -4.04, p < 0.001), statistically indistinguishable from ELIZA (z = 0.050, p = 0.96).
  • Interrogators relied on style over substance: 43% of stated reasons concerned linguistic style, 24% socio-emotional factors, and 10% knowledge and reasoning. Strategies were 36% small talk, 25% social & emotional, 13% knowledge & reasoning; accuracy was higher for questions about human experiences (75%), logic & math (78%), and accusations of being an AI (80%, though possibly a consequence rather than a cause of correct verdicts).

Methods

  • Randomized, controlled, preregistered two-party Turing test: a single interrogator conversed with a single witness (human or AI) for up to 5 minutes via a messaging-app interface, then issued a binary verdict with a confidence slider and free-text reasoning. AI witnesses were GPT-4, GPT-3.5, and ELIZA, using a hand-crafted "tiktok" persona prompt; messages were delayed to mimic typing and reading time.
  • Main study: 402 interrogators recruited via Prolific (168 female, 212 male, 9 nonbinary, 13 prefer not to say; mean age 35, σ = 11.2), targeting ~100 games per condition after exclusions (101 Human, 101 GPT-3.5, 100 GPT-4, 100 ELIZA). Confirmatory analyses used logistic regressions in R plus a Bayesian model (brms).
  • Replication: 111 undergraduate participants from an institutional subject pool, each completing 8 rounds, generating 629 games (408 with AI witnesses, 221 with human witnesses); GPT-4o was used with 10 prompts including the original tiktok prompt and ELIZA as baseline. Strategy and reason tagging was done by two authors with multi-label classification (agreement 0.67 for strategies, 0.49 for reasons).

Limitations

  • The paper states that the strategy-effectiveness data are purely correlational, noting that accusations of being an AI tended to precede correct identifications; it also notes the qinn prompt's 86.7% pass rate came from only 15 games and is unreliable, and that GPT-4o's parity with humans was not preregistered and would need confirmation in follow-up work.
  • The paper acknowledges that the 50% baseline criterion involves confirming a null hypothesis (a system could be indistinguishable from chance because participants guessed randomly or because the study was underpowered), and that one could always argue a more stringent baseline than ELIZA is needed.
  • The paper raises the question of credit attribution: since the minimal prompt achieved only 25% versus the main prompt's 76%, it is unclear to what extent success should be attributed to the model itself versus careful prompting. Evident from the text: the main study could not share transcripts (participants did not consent), so only replication transcripts were released.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3715275.3732108

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

13 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Figures

Figure 1 A selection of conversations between human interrogators (green) and witnesses (grey). One of these four conversations is with a human witness, the rest are with AI. Interrogator verdicts and ground truth identities are below. 1
Figure 1 A selection of conversations between human interrogators (green) and witnesses (grey). One of these four conversations is with a human witness, the rest are with AI. Interrogator verdicts and ground truth identities are below. 1CC-BY · From the PDF
Figure 2 Pass rates (left) and interrogator confidence (right) for each witness type. Left: Pass rates are the proportion of the time a witness type was judged to be human. Error bars represent 95% bootstrap confidence intervals. Significance stars above each bar indicate whether the pass rate was significantly different from 50%. Comparisons show significant differences in pass rates between witness types. Right: Confidence in human and AI judgements for each witness type. Each point represents a single game. Points further toward the left and right indicate higher confidence in AI and human verdicts respectively.
Figure 2 Pass rates (left) and interrogator confidence (right) for each witness type. Left: Pass rates are the proportion of the time a witness type was judged to be human. Error bars represent 95% bootstrap confidence intervals. Significance stars above each bar indicate whether the pass rate was significantly different from 50%. Comparisons show significant differences in pass rates between witness types. Right: Confidence in human and AI judgements for each witness type. Each point represents a single game. Points further toward the left and right indicate higher confidence in AI and human verdicts respectively.CC-BY · From the PDF
Figure 3 Classification of strategies employed by interrogators by proportion of games (left) and mean accuracy of games where strategies were deployed (right). Participants often engaged in small talk, asking witnesses about their personal details, activities, or opinions. Interrogators who asked about logic, current events, or human emotions and experiences tended to be more accurate.
Figure 3 Classification of strategies employed by interrogators by proportion of games (left) and mean accuracy of games where strategies were deployed (right). Participants often engaged in small talk, asking witnesses about their personal details, activities, or opinions. Interrogators who asked about logic, current events, or human emotions and experiences tended to be more accurate.CC-BY · From the PDF
Figure 4 Proportion of interrogator reasons for AI verdicts (left) and Human verdicts (right), excluding ELIZA games. In both cases, interrogators were much more likely to cite linguistic style or socio-emotional factors such as personality, rather than factors more traditionally associated with intelligence, such as knowledge and reasoning.
Figure 4 Proportion of interrogator reasons for AI verdicts (left) and Human verdicts (right), excluding ELIZA games. In both cases, interrogators were much more likely to cite linguistic style or socio-emotional factors such as personality, rather than factors more traditionally associated with intelligence, such as knowledge and reasoning.CC-BY · From the PDF
Figure 5 Pass rates (left) and interrogator confidence (right) for the replication study. Significance stars above each bar indicate whether the pass rate was significantly different from 50%. Comparisons show significant differences in pass rates between witness types. Right: Confidence in human and AI judgements for each witness type. Each point represents a single game. Points further toward the left and right indicate higher confidence in AI and human verdicts respectively.
Figure 5 Pass rates (left) and interrogator confidence (right) for the replication study. Significance stars above each bar indicate whether the pass rate was significantly different from 50%. Comparisons show significant differences in pass rates between witness types. Right: Confidence in human and AI judgements for each witness type. Each point represents a single game. Points further toward the left and right indicate higher confidence in AI and human verdicts respectively.CC-BY · From the PDF
Figure 6 Turing test game interface. Left: an in-progress conversation between an interrogator (green) and a witness (grey). The timer at the top shows time remaining in the game. Right: the decision interface the interrogator uses to give their verdict.
Figure 6 Turing test game interface. Left: an in-progress conversation between an interrogator (green) and a witness (grey). The timer at the top shows time remaining in the game. Right: the decision interface the interrogator uses to give their verdict.CC-BY · From the PDF
Figure 7 Bayesian estimates of pass rates for each witness type. Each density plot represents draws from the posterior distribution of a Bayesian model estimating pass rates for each witness type. A pass rate of 0.5 was above the 95% credible interval for ELIZA and below the interval for humans, but within this internal for GPT-4 and GPT-3.5.
Figure 7 Bayesian estimates of pass rates for each witness type. Each density plot represents draws from the posterior distribution of a Bayesian model estimating pass rates for each witness type. A pass rate of 0.5 was above the 95% credible interval for ELIZA and below the interval for humans, but within this internal for GPT-4 and GPT-3.5.CC-BY · From the PDF
Figure 8 Confidence calibration by witness type. Interrogators were well calibrated for ELIZA, with higher confidence correlating with higher accuracy. This trend was less pronounced for GPT models and reversed for Human witnesses.
Figure 8 Confidence calibration by witness type. Interrogators were well calibrated for ELIZA, with higher confidence correlating with higher accuracy. This trend was less pronounced for GPT models and reversed for Human witnesses.CC-BY · From the PDF
Figure 9 Distribution of demographic data across 400 interrogators. Interrogators tended to be undergraduate educated, in their 20s-30s, have some knowledge about LLMs and interact with chatbots at least once a month.
Figure 9 Distribution of demographic data across 400 interrogators. Interrogators tended to be undergraduate educated, in their 20s-30s, have some knowledge about LLMs and interact with chatbots at least once a month.CC-BY · From the PDF
Figure 10 Accuracy by demographic factors. Accuracy is 1 when the interrogator correctly guesses the identity of the witness and 0 otherwise. Only age was a significant predictor of accuracy.
Figure 10 Accuracy by demographic factors. Accuracy is 1 when the interrogator correctly guesses the identity of the witness and 0 otherwise. Only age was a significant predictor of accuracy.CC-BY · From the PDF
Figure 11 All strategy tags by category.
Figure 11 All strategy tags by category.CC-BY · From the PDF
Figure 12 All reason tags for AI verdicts by category, for all witness types.
Figure 12 All reason tags for AI verdicts by category, for all witness types.CC-BY · From the PDF
Figure 13 All reason tags for Human verdicts by category, for all witness types.
Figure 13 All reason tags for Human verdicts by category, for all witness types.CC-BY · From the PDF
Figure 14 Top 10 reason tags for human and GPT-4 witnesses, by verdict and witness type.
Figure 14 Top 10 reason tags for human and GPT-4 witnesses, by verdict and witness type.CC-BY · From the PDF
Figure 15 Top reason classes for ELIZA
Figure 15 Top reason classes for ELIZACC-BY · From the PDF

Verification

Claims the paper makes

8 Valid · 2 Not valid · 4 Not checked · 116 Own results and interpretations

67 more claims

Create a free account to see all 130 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 55
  • It may state two separate claims as one. 15
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

2 of 2

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upz = 2.42reported p = .017 · recomputed p = .016

    “(z = 2.42, p = .017”

  • Doesn’t add upz = -0.023reported p = .869 · recomputed p = .982

    “(z = -0.023,p = .869”

  • Consistentz = 0.799reported p = .424 · recomputed p = .424

    “z = 0.799, p = .424”

  • Consistentz = -5.243reported p < .001 · recomputed p < .001

    “z = -5.243,p < .001”

  • Consistentz = 4.11reported p < .001 · recomputed p < .001

    “(z = 4.11, p < .001”

5 of 13 · the rest aren’t loaded

8 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 36 references

Research works printed but not linked: 4

  1. European Parliament and Council. 2024. Regulation (EU) 2024/1183—Artificial Intelligence Act. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L:2024:206:FULL Official Journal of the European Union L 206/1.
  2. Patrick Hayes and Kenneth Ford. 1995. Turing Test Considered Harmful. IJCAI 1 (1995) 972–977.
  3. Graham Oppy and David Dowe. 2021. The Turing Test. In The Stanford Encyclopedia of Philosophy (winter 2021 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
  4. Vishvesh Soni. 2023. Large Language Models for Enhancing Customer Lifecycle Management. Journal of Empirical Social Science Studies 7 1 (Feb. 2023) 67–89.

Cited by

See all 76

Metadata

Authors
Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K.
Affiliations
University of California San Diego (US)
Venue
Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2025-06-23
Pages
1615-1639
OpenAlex
W4411541869
Cited by
76
References
36
Citations by year
2026: 38 · 2025: 32
FWCI
66.38
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Open Philanthropy Project
  • University of California, San Diego
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
Turing test, GPT-4, GPT-4o, Eliza, GPT-3.5, human-AI interaction, AI deception, conversational AI, machine intelligence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: People cannot distinguish GPT-4 from a human in a Turing test. https://app.cochat.ai/papers/10.1145/3715275.3732108
The paper
Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K. (2025). People cannot distinguish GPT-4 from a human in a Turing test. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3715275.3732108

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.