The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index; 18 of the 29 sources its claims cite were checked for support.
People cannot distinguish GPT-4 from a human in a Turing test
Open accessAbstract
AI systems that can fool people into thinking that they are human could have widespread social and economic consequences.In order to measure this ability, we evaluated 3 systems (ELIZA, GPT-3.5 and GPT-4) in a randomized, controlled, and preregistered Turing test.Human participants had a 5 minute conversation with either a human or an AI, and judged whether or not they thought their interlocutor was human.GPT-4 was judged to be a human 54% of the time, significantly more often than ELIZA (22%) but less often than actual humans (67%).In order to test the generalizability of our results, we replicated the study on a second population (undergraduate students) and found that the same prompt with GPT-4o achieved a pass rate of 77%, slightly higher than the human pass rate of 71%.On some interpretations, the results provide the first robust empirical demonstration that any artificial system passes an interactive 2-player Turing test.The results have implications for debates around machine intelligence and, more urgently, suggest that deception by current AI systems may go undetected.Analysis of participants' strategies and reasoning suggests that stylistic and socio-emotional factors play a larger role in passing the Turing test than traditional notions of intelligence.We release the full transcripts of the replication data to enable further investigation of human-AI interaction dynamics and deception.
Overview
CoChat Summary
TL;DR
In a preregistered, randomized two-party Turing test, human interrogators judged GPT-4 to be human 54% of the time—statistically indistinguishable from chance—while judging actual humans human 67% of the time and ELIZA only 22%. A conceptual replication with GPT-4o on an undergraduate sample found a 76% pass rate, slightly above the 71% human pass rate, suggesting stylistic and socio-emotional cues matter more than traditional notions of intelligence.
Key findings
- Main study pass rates: GPT-4 54%, GPT-3.5 50%, ELIZA 22%, humans 67%. GPT-4's pass rate was not significantly different from chance (z = 0.799, p = 0.424), while ELIZA's was significantly below chance (z = -5.243, p < 0.001), serving as a manipulation check.
- GPT-4's pass rate was significantly higher than ELIZA's (z = 4.11, p < 0.001) and significantly lower than humans' (z = 2.42, p = 0.017), but did not differ from GPT-3.5 (z = 0.497, p = 0.619). Bayesian estimates put GPT-4's pass rate between 44% and 63% and ELIZA's between 15% and 31%.
- Replication (GPT-4o, tiktok prompt): 76% pass rate across 63 games, significantly above chance (z = 3.93, p < 0.001) and above ELIZA (z = 5.47, p < 0.001), but not significantly different from human witnesses at 71% (z = -0.735, p = 0.462). A minimal prompt achieved only 25% (z = -4.04, p < 0.001), statistically indistinguishable from ELIZA (z = 0.050, p = 0.96).
- Interrogators relied on style over substance: 43% of stated reasons concerned linguistic style, 24% socio-emotional factors, and 10% knowledge and reasoning. Strategies were 36% small talk, 25% social & emotional, 13% knowledge & reasoning; accuracy was higher for questions about human experiences (75%), logic & math (78%), and accusations of being an AI (80%, though possibly a consequence rather than a cause of correct verdicts).
Methods
- Randomized, controlled, preregistered two-party Turing test: a single interrogator conversed with a single witness (human or AI) for up to 5 minutes via a messaging-app interface, then issued a binary verdict with a confidence slider and free-text reasoning. AI witnesses were GPT-4, GPT-3.5, and ELIZA, using a hand-crafted "tiktok" persona prompt; messages were delayed to mimic typing and reading time.
- Main study: 402 interrogators recruited via Prolific (168 female, 212 male, 9 nonbinary, 13 prefer not to say; mean age 35, σ = 11.2), targeting ~100 games per condition after exclusions (101 Human, 101 GPT-3.5, 100 GPT-4, 100 ELIZA). Confirmatory analyses used logistic regressions in R plus a Bayesian model (brms).
- Replication: 111 undergraduate participants from an institutional subject pool, each completing 8 rounds, generating 629 games (408 with AI witnesses, 221 with human witnesses); GPT-4o was used with 10 prompts including the original tiktok prompt and ELIZA as baseline. Strategy and reason tagging was done by two authors with multi-label classification (agreement 0.67 for strategies, 0.49 for reasons).
Limitations
- The paper states that the strategy-effectiveness data are purely correlational, noting that accusations of being an AI tended to precede correct identifications; it also notes the qinn prompt's 86.7% pass rate came from only 15 games and is unreliable, and that GPT-4o's parity with humans was not preregistered and would need confirmation in follow-up work.
- The paper acknowledges that the 50% baseline criterion involves confirming a null hypothesis (a system could be indistinguishable from chance because participants guessed randomly or because the study was underpowered), and that one could always argue a more stringent baseline than ELIZA is needed.
- The paper raises the question of credit attribution: since the minimal prompt achieved only 25% versus the main prompt's 76%, it is unclear to what extent success should be attributed to the model itself versus careful prompting. Evident from the text: the main study could not share transcripts (participants did not consent), so only replication transcripts were released.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
8 Valid · 2 Not valid · 4 Not checked · 116 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“A recent large-scale study [21] found that humans were 60% accurate in identifying a range of modern language models in two-minute online conversations.”
§ People cannot distinguish GPT-4 from a human in a Turing test > 1 Introduction 1.The Turing test
What the sources say
- Supportedcites [21] 21. Daniel Jannai, Amos Meron, Barak Lenz, Yoav Levine, and Yoav Shoham. 2023. Human or Not? A Gamified Approach to the Turing Test. arXiv:2305.20010 [cs] source
The Introduction reports more than 10 million conversations and says users correctly identified AI bots in 60% of games involving a bot, with conversations limited to two minutes. Section 2.2 says the bots used several language models, including Jurassic-2, GPT-4, and Cohere. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“one of the central novel risks of LLMs is their ability to interact dynamically with users [7,23].”
§ People cannot distinguish GPT-4 from a human in a Turing test > 1 Introduction 1.The Turing test
What the sources say
- Supportedcites [7] 7. Matthew Burtell and Thomas Woodside. 2023. Artificial Influence: An Analysis Of AI-Driven Persuasion. arXiv:2303.08721 [cs] source
Section 2.1 identifies sustained one-on-one interaction as a way an AI agent can build familiarity and persuade a user. Section 2.2 illustrates the risk with a chatbot that steers a conversation using messages tailored to the recipient, unlike conventional attack ads. (LLM, conf=0.88)
- Supportedcites [23] 23. Cameron R. Jones and Benjamin K. Bergen. 2024. Lies, Damned Lies, and Distri- butional Language Statistics: Persuasion and Deception with Large Language Models. doi:10.48550/arXiv.2412.17128 arXiv:2412.17128 [cs] source
Section 3.4.3, “Interacting with humans” (page not provided), calls dynamic interaction with a human interlocutor one of the most novel and potentially impactful aspects of LLMs. The same section reports that such interactions can significantly affect people’s beliefs and actions, supporting the citing sentence’s characterization of this ability as a risk. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“significantly more often than ELIZA (22%) but less often than actual humans (67%)”
- Also stated in: § 3.1 Pass rates · plain
“the ELIZA baseline (22%)”
§ People cannot distinguish GPT-4 from a human in a Turing test > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.1 Pass rates · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 was judged to be a human 54% of the time, significantly more often than ELIZA (22%) but less often than actual humans (67%).”
- Also stated in: § 3.1 Pass rates · plain
“GPT-4 achieved a pass rate of 54%, surpassing GPT-3.5 (50%) and the ELIZA baseline (22%), but lagging behind human participants (67%, see Figure 2).”
§ People cannot distinguish GPT-4 from a human in a Turing test > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.1 Pass rates · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“less often than actual humans (67%)”
- Also stated in: § 3.1 Pass rates · plain
“human participants (67%, see Figure 2)”
§ People cannot distinguish GPT-4 from a human in a Turing test > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.1 Pass rates · plain
67 more claims
Create a free account to see all 130 claims and every source verdict.
- Probably not a substantial claim. 55
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“(𝑧 = 2.42, 𝑝 = 0.017”
§ People cannot distinguish GPT-4 from a human in a Turing test > 3.1 Pass rates
Reported p = .017 · recomputed p = .016
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“(𝑧 = -0.023,𝑝 = 0.869”
§ People cannot distinguish GPT-4 from a human in a Turing test > 3.1 Pass rates
Reported p = .869 · recomputed p = .982
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upz = 2.42reported p = .017 · recomputed p = .016
“(z = 2.42, p = .017”
- Doesn’t add upz = -0.023reported p = .869 · recomputed p = .982
“(z = -0.023,p = .869”
- Consistentz = 0.799reported p = .424 · recomputed p = .424
“z = 0.799, p = .424”
- Consistentz = -5.243reported p < .001 · recomputed p < .001
“z = -5.243,p < .001”
- Consistentz = 4.11reported p < .001 · recomputed p < .001
“(z = 4.11, p < .001”
8 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ People cannot distinguish GPT-4 from a human in a Turing test > 4.1 Methods
“In addition to the prompt used in the original study, we included a selection of 9 other prompts.”
§ People cannot distinguish GPT-4 from a human in a Turing test > 4.1 Methods
“The 4 high-priority AI witnesses were tiktok (the prompt used in the the original study above), minimal (a minimalistic prompt that simply stated that the model was a witness in a Turing test and should try to convince the interrogator it was human), sardonic (a prompt encouraging the model to be sardonic), and the eliza baseline.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ People cannot distinguish GPT-4 from a human in a Turing test > 1 Introduction 1.The Turing test
“We replicated our study on a second defined population (undergraduate students), and found that the same modelprompt combination achieved a pass rate of 77%, slightly higher than the human pass rate of 71%.”
§ People cannot distinguish GPT-4 from a human in a Turing test > 4 Replication
“First, we used GPT-4o [36]-rather than GPT-4-which had become OpenAI's default model in their ChatGPT service.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ People cannot distinguish GPT-4 from a human in a Turing test > Abstract
“the same prompt with GPT-4o achieved a pass rate of 77%, slightly higher than the human pass rate of 71%”
§ People cannot distinguish GPT-4 from a human in a Turing test > 4.2 Results
“GPT-4o's pass rate was 76% across the 63 games in which it was a witness”
Citations
Cites
Artificial intelligence: a modern approach
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 21,974
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 2×
“Some have objected that the test is too hard [13] or too chauvu-nistic [45], however, these concerns are less pressing if a system does appear to pass [51].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“First, studies involving human subjects tend to have nontrivial rates of replication failure, and so replication is always good practice [8, 32].”
Searle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954 · cited 1×
“Interrogators were more likely to be accurate when they asked about human experiences (75%), logic & math (78%), or accused the witness of being an AI (80%), though the latter may result from accusations preceding an accurate verdict rather than the effectiveness of the strategy.”
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 3×
“The Turing test has a storied history, having “unquestionably generated more commentary and controversy than any other article in the field of artificial intelligence” [13, p. 116].”
InfluentialGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Advanced Bayesian Multilevel Modeling with the R Package brms
Bürkner, P.-C. (2018) · The R Journal · doi:10.32614/rj-2018-017 · Cited by 3,015 · cited 1×
“These judgments tell us as much about interrogators’ evolving mental models of AI systems as they do about the systems themselves—suggesting that as AI capabilities advance, the very notion of what constitutes "human-like" behavior may shift in unexpected ways.”
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · cited 1×
“First, studies involving human subjects tend to have nontrivial rates of replication failure, and so replication is always good practice [8, 32].”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 1×
“Interrogators were more likely to be accurate when they asked about human experiences (75%), logic & math (78%), or accused the witness of being an AI (80%), though the latter may result from accusations preceding an accurate verdict rather than the effectiveness of the strategy.”
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Research works printed but not linked: 4
- European Parliament and Council. 2024. Regulation (EU) 2024/1183—Artificial Intelligence Act. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L:2024:206:FULL Official Journal of the European Union L 206/1.
- Patrick Hayes and Kenneth Ford. 1995. Turing Test Considered Harmful. IJCAI 1 (1995) 972–977.
- Graham Oppy and David Dowe. 2021. The Turing Test. In The Stanford Encyclopedia of Philosophy (winter 2021 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
- Vishvesh Soni. 2023. Large Language Models for Enhancing Customer Lifecycle Management. Journal of Empirical Social Science Studies 7 1 (Feb. 2023) 67–89.
Cited by
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
There is no such thing as conscious artificial intelligence
Porębski, A., & Figura, J. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05868-8 · Cited by 20
“Indeed, according to the initial fi ndings, some LLMs passed this test (Jones et al., 2025; Jones and Bergen, 2025).”
Larooij, M., & Törnberg, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11412-6 · Cited by 52
“Moreover, generative agents are capable of producing and interpreting natural language in ways that can be indistinguishable from human communication (Jones et al. 2025).”
Duan, J., Li, W., Bai, Q., Nguyen, M., Wang, X., & Jiang, J. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3545822 · Cited by 19
“However, it still falls short of actual humans, who were identified as human 67% of the time [9].”
Savage, T., Ma, S. P., Boukil, A., Rangan, E. S., Patel, V., López, I., & Chen, J. H. (2025) · Journal of Medical Internet Research · doi:10.2196/76048 · Cited by 24
Metadata
- Authors
- Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K.
- Affiliations
- University of California San Diego (US)
- Venue
- Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2025-06-23
- Pages
- 1615-1639
- OpenAlex
- W4411541869
- Semantic Scholar
- 505c7b471457ac447efdb9abc714247dd10dbef8
- Cited by
- 76
- References
- 36
- Citations by year
- 2026: 38 · 2025: 32
- FWCI
- 66.38
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- University of California, San Diego
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- Turing test, GPT-4, GPT-4o, Eliza, GPT-3.5, human-AI interaction, AI deception, conversational AI, machine intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: People cannot distinguish GPT-4 from a human in a Turing test. https://app.cochat.ai/papers/10.1145/3715275.3732108
- The paper
- Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K. (2025). People cannot distinguish GPT-4 from a human in a Turing test. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3715275.3732108
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.