People cannot distinguish GPT-4 from a human in a Turing test
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index; 18 of the 29 sources its claims cite were checked for support.
Abstract
AI systems that can fool people into thinking that they are human could have widespread social and economic consequences.In order to measure this ability, we evaluated 3 systems (ELIZA, GPT-3.5 and GPT-4) in a randomized, controlled, and preregistered Turing test.Human participants had a 5 minute conversation with either a human or an AI, and judged whether or not they thought their interlocutor was human.GPT-4 was judged to be a human 54% of the time, significantly more often than ELIZA (22%) but less often than actual humans (67%).In order to test the generalizability of our results, we replicated the study on a second population (undergraduate students) and found that the same prompt with GPT-4o achieved a pass rate of 77%, slightly higher than the human pass rate of 71%.On some interpretations, the results provide the first robust empirical demonstration that any artificial system passes an interactive 2-player Turing test.The results have implications for debates around machine intelligence and, more urgently, suggest that deception by current AI systems may go undetected.Analysis of participants' strategies and reasoning suggests that stylistic and socio-emotional factors play a larger role in passing the Turing test than traditional notions of intelligence.We release the full transcripts of the replication data to enable further investigation of human-AI interaction dynamics and deception.
Overview
CoChat Summary
TL;DR
In a preregistered, randomized two-party Turing test, human interrogators judged GPT-4 to be human 54% of the time—statistically indistinguishable from chance—while judging actual humans human 67% of the time and ELIZA only 22%. A conceptual replication with GPT-4o on an undergraduate sample found a 76% pass rate, slightly above the 71% human pass rate, suggesting stylistic and socio-emotional cues matter more than traditional notions of intelligence.
Key findings
- Main study pass rates: GPT-4 54%, GPT-3.5 50%, ELIZA 22%, humans 67%. GPT-4's pass rate was not significantly different from chance (z = 0.799, p = 0.424), while ELIZA's was significantly below chance (z = -5.243, p < 0.001), serving as a manipulation check.
- GPT-4's pass rate was significantly higher than ELIZA's (z = 4.11, p < 0.001) and significantly lower than humans' (z = 2.42, p = 0.017), but did not differ from GPT-3.5 (z = 0.497, p = 0.619). Bayesian estimates put GPT-4's pass rate between 44% and 63% and ELIZA's between 15% and 31%.
- Replication (GPT-4o, tiktok prompt): 76% pass rate across 63 games, significantly above chance (z = 3.93, p < 0.001) and above ELIZA (z = 5.47, p < 0.001), but not significantly different from human witnesses at 71% (z = -0.735, p = 0.462). A minimal prompt achieved only 25% (z = -4.04, p < 0.001), statistically indistinguishable from ELIZA (z = 0.050, p = 0.96).
- Interrogators relied on style over substance: 43% of stated reasons concerned linguistic style, 24% socio-emotional factors, and 10% knowledge and reasoning. Strategies were 36% small talk, 25% social & emotional, 13% knowledge & reasoning; accuracy was higher for questions about human experiences (75%), logic & math (78%), and accusations of being an AI (80%, though possibly a consequence rather than a cause of correct verdicts).
Methods
- Randomized, controlled, preregistered two-party Turing test: a single interrogator conversed with a single witness (human or AI) for up to 5 minutes via a messaging-app interface, then issued a binary verdict with a confidence slider and free-text reasoning. AI witnesses were GPT-4, GPT-3.5, and ELIZA, using a hand-crafted "tiktok" persona prompt; messages were delayed to mimic typing and reading time.
- Main study: 402 interrogators recruited via Prolific (168 female, 212 male, 9 nonbinary, 13 prefer not to say; mean age 35, σ = 11.2), targeting ~100 games per condition after exclusions (101 Human, 101 GPT-3.5, 100 GPT-4, 100 ELIZA). Confirmatory analyses used logistic regressions in R plus a Bayesian model (brms).
- Replication: 111 undergraduate participants from an institutional subject pool, each completing 8 rounds, generating 629 games (408 with AI witnesses, 221 with human witnesses); GPT-4o was used with 10 prompts including the original tiktok prompt and ELIZA as baseline. Strategy and reason tagging was done by two authors with multi-label classification (agreement 0.67 for strategies, 0.49 for reasons).
Limitations
- The paper states that the strategy-effectiveness data are purely correlational, noting that accusations of being an AI tended to precede correct identifications; it also notes the qinn prompt's 86.7% pass rate came from only 15 games and is unreliable, and that GPT-4o's parity with humans was not preregistered and would need confirmation in follow-up work.
- The paper acknowledges that the 50% baseline criterion involves confirming a null hypothesis (a system could be indistinguishable from chance because participants guessed randomly or because the study was underpowered), and that one could always argue a more stringent baseline than ELIZA is needed.
- The paper raises the question of credit attribution: since the minimal prompt achieved only 25% versus the main prompt's 76%, it is unclear to what extent success should be attributed to the model itself versus careful prompting. Evident from the text: the main study could not share transcripts (participants did not consent), so only replication transcripts were released.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
8 Valid · 2 Not valid · 4 Not checked · 116 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“A recent large-scale study [21] found that humans were 60% accurate in identifying a range of modern language models in two-minute online conversations.”
§ People cannot distinguish GPT-4 from a human in a Turing test > 1 Introduction 1.The Turing test
What the sources say
- Supportedcites [21] 21. Daniel Jannai, Amos Meron, Barak Lenz, Yoav Levine, and Yoav Shoham. 2023. Human or Not? A Gamified Approach to the Turing Test. arXiv:2305.20010 [cs] source
The Introduction reports more than 10 million conversations and says users correctly identified AI bots in 60% of games involving a bot, with conversations limited to two minutes. Section 2.2 says the bots used several language models, including Jurassic-2, GPT-4, and Cohere. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“one of the central novel risks of LLMs is their ability to interact dynamically with users [7,23].”
§ People cannot distinguish GPT-4 from a human in a Turing test > 1 Introduction 1.The Turing test
What the sources say
- Supportedcites [7] 7. Matthew Burtell and Thomas Woodside. 2023. Artificial Influence: An Analysis Of AI-Driven Persuasion. arXiv:2303.08721 [cs] source
Section 2.1 identifies sustained one-on-one interaction as a way an AI agent can build familiarity and persuade a user. Section 2.2 illustrates the risk with a chatbot that steers a conversation using messages tailored to the recipient, unlike conventional attack ads. (LLM, conf=0.88)
- Supportedcites [23] 23. Cameron R. Jones and Benjamin K. Bergen. 2024. Lies, Damned Lies, and Distri- butional Language Statistics: Persuasion and Deception with Large Language Models. doi:10.48550/arXiv.2412.17128 arXiv:2412.17128 [cs] source
Section 3.4.3, “Interacting with humans” (page not provided), calls dynamic interaction with a human interlocutor one of the most novel and potentially impactful aspects of LLMs. The same section reports that such interactions can significantly affect people’s beliefs and actions, supporting the citing sentence’s characterization of this ability as a risk. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“significantly more often than ELIZA (22%) but less often than actual humans (67%)”
- Also stated in: § 3.1 Pass rates · plain
“the ELIZA baseline (22%)”
§ People cannot distinguish GPT-4 from a human in a Turing test > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.1 Pass rates · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 was judged to be a human 54% of the time, significantly more often than ELIZA (22%) but less often than actual humans (67%).”
- Also stated in: § 3.1 Pass rates · plain
“GPT-4 achieved a pass rate of 54%, surpassing GPT-3.5 (50%) and the ELIZA baseline (22%), but lagging behind human participants (67%, see Figure 2).”
§ People cannot distinguish GPT-4 from a human in a Turing test > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.1 Pass rates · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“less often than actual humans (67%)”
- Also stated in: § 3.1 Pass rates · plain
“human participants (67%, see Figure 2)”
§ People cannot distinguish GPT-4 from a human in a Turing test > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.1 Pass rates · plain
67 more claims
Create a free account to see all 130 claims and every source verdict.
- Probably not a substantial claim. 55
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom, each partial η² from its F, and each ratio, mean difference or B printed with a p checked against its own confidence interval.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“(𝑧 = 2.42, 𝑝 = 0.017”
§ People cannot distinguish GPT-4 from a human in a Turing test > 3.1 Pass rates
Reported p = .017 · recomputed p = .016
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“(𝑧 = -0.023,𝑝 = 0.869”
§ People cannot distinguish GPT-4 from a human in a Turing test > 3.1 Pass rates
Reported p = .869 · recomputed p = .982
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upz = 2.42reported p = .017 · recomputed p = .016
“(z = 2.42, p = .017”
- Doesn’t add upz = -0.023reported p = .869 · recomputed p = .982
“(z = -0.023,p = .869”
- Consistentz = 0.799reported p = .424 · recomputed p = .424
“z = 0.799, p = .424”
- Consistentz = -5.243reported p < .001 · recomputed p < .001
“z = -5.243,p < .001”
- Consistentz = 4.11reported p < .001 · recomputed p < .001
“(z = 4.11, p < .001”
8 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ People cannot distinguish GPT-4 from a human in a Turing test > 4.1 Methods
“In addition to the prompt used in the original study, we included a selection of 9 other prompts.”
§ People cannot distinguish GPT-4 from a human in a Turing test > 4.1 Methods
“The 4 high-priority AI witnesses were tiktok (the prompt used in the the original study above), minimal (a minimalistic prompt that simply stated that the model was a witness in a Turing test and should try to convince the interrogator it was human), sardonic (a prompt encouraging the model to be sardonic), and the eliza baseline.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ People cannot distinguish GPT-4 from a human in a Turing test > 1 Introduction 1.The Turing test
“We replicated our study on a second defined population (undergraduate students), and found that the same modelprompt combination achieved a pass rate of 77%, slightly higher than the human pass rate of 71%.”
§ People cannot distinguish GPT-4 from a human in a Turing test > 4 Replication
“First, we used GPT-4o [36]-rather than GPT-4-which had become OpenAI's default model in their ChatGPT service.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ People cannot distinguish GPT-4 from a human in a Turing test > Abstract
“the same prompt with GPT-4o achieved a pass rate of 77%, slightly higher than the human pass rate of 71%”
§ People cannot distinguish GPT-4 from a human in a Turing test > 4.2 Results
“GPT-4o's pass rate was 76% across the 63 games in which it was a witness”
Citations
Cites
Artificial intelligence: a modern approach
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 21,974
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 2×
“Some have objected that the test is too hard [13] or too chauvu-nistic [45], however, these concerns are less pressing if a system does appear to pass [51].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“First, studies involving human subjects tend to have nontrivial rates of replication failure, and so replication is always good practice [8, 32].”
Searle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954 · cited 1×
“Interrogators were more likely to be accurate when they asked about human experiences (75%), logic & math (78%), or accused the witness of being an AI (80%), though the latter may result from accusations preceding an accurate verdict rather than the effectiveness of the strategy.”
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 3×
“The Turing test has a storied history, having “unquestionably generated more commentary and controversy than any other article in the field of artificial intelligence” [13, p. 116].”
InfluentialGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Advanced Bayesian Multilevel Modeling with the R Package brms
Bürkner, P.-C. (2018) · The R Journal · doi:10.32614/rj-2018-017 · Cited by 3,015 · cited 1×
“These judgments tell us as much about interrogators’ evolving mental models of AI systems as they do about the systems themselves—suggesting that as AI capabilities advance, the very notion of what constitutes "human-like" behavior may shift in unexpected ways.”
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · cited 1×
“First, studies involving human subjects tend to have nontrivial rates of replication failure, and so replication is always good practice [8, 32].”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 1×
“Interrogators were more likely to be accurate when they asked about human experiences (75%), logic & math (78%), or accused the witness of being an AI (80%), though the latter may result from accusations preceding an accurate verdict rather than the effectiveness of the strategy.”
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Defending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569 · cited 2×
“AI systems that can reliably masquerade as people can be used by criminals for fraud [1, 14, 59], by governments for foreign influence [26, 35], and by commercial entities to create engaging but ultimately manipulative experiences for users [7, 57].”
Block, N. (1981) · The Philosophical Review · doi:10.2307/2184371 · Cited by 562 · cited 1×
“Still others have argued that no behavioural test can measure intelligence; that intelligence relies upon the right kind of inner mechanism or causal relationship with the world [2, 4, 46] (however, see recent philosophical treatments of the potential for LLMs to meet these criteria [15, 29, 39]).”
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · cited 1×
“Lastly, these risks are likely to be unevenly distributed across the population, with systems best able to exploit the most vulnerable [30, 54].”
Technology, autonomy, and manipulation
Susser, D., Roessler, B., & Nissenbaum, H. (2019) · Internet Policy Review · doi:10.14763/2019.2.1410 · Cited by 474
Pinar Saygin, A., Cicekli, I., & Akman, V. (2000) · Minds and Machines · doi:10.1023/a:1011288000451 · Cited by 383 · cited 4×
“When users engage in direct, real-time conversation with a suspected AI system, they can employ strategic questioning and adapt their approach based on responses.”
InfluentialThe debate over understanding in AI’s large language models
Mitchell, M., & Krakauer, D. C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2215907120 · Cited by 340 · cited 1×
“At the start of each game, several additional pieces of information were appended to the prompt, including a suggested name, location, languages spoken, and timestamps for the model.”
The Turing Test: the first 50 years
French, R. M. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01453-4 · Cited by 250 · cited 6×
“When users engage in direct, real-time conversation with a suspected AI system, they can employ strategic questioning and adapt their approach based on responses.”
InfluentialAI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243 · cited 2×
“In order to enable other researchers to scrutinize our data, and to investigate other questions for which this data could be helpful, we sought approval to share full transcripts from the replication study (after any potentially sensitive or personally identifiable information had been removed).”
AI Deception: A Survey of Examples, Risks, and Potential Solutions
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2308.14752 · Cited by 243 · cited 2×
“In order to enable other researchers to scrutinize our data, and to investigate other questions for which this data could be helpful, we sought approval to share full transcripts from the replication study (after any potentially sensitive or personally identifiable information had been removed).”
Perkins, M., Roe, J., Postma, D., McGaughran, J., & Hickerson, D. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-023-09492-6 · Cited by 234
Game of Tones: Faculty detection of GPT-4 generated content in university assessments
Perkins, M., Roe, J., Postma, D., McGaughran, J., & Hickerson, D. (2023) · arXiv · doi:10.48550/arxiv.2305.18081 · Cited by 233
Lessons from a Restricted Turing Test
Shieber, S. M. (1994) · arXiv · doi:10.48550/arxiv.cmp-lg/9404002 · Cited by 222
A Watermark for Large Language Models
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023) · arXiv · doi:10.48550/arxiv.2301.10226 · Cited by 194
OpenAI, :, Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., et al. (2024) · arXiv · Cited by 180
AI and the Everything in the Whole Wide World Benchmark
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021) · arXiv · doi:10.48550/arxiv.2111.15366 · Cited by 127 · cited 1×
“AI verdicts in ELIZA’s games predominantly cited the model’s unnatural responses and lack of contextual awareness (see Figure 15).”
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · cited 1×
“Large Language Models like GPT-4 [ While these advances offer some potential advantages—allowing systems to play valuable societal roles which have until now been the sole preserve of people—they also bring attendant risks [11, 18, 23, 38].”
Self-citationDoes GPT-4 pass the Turing test?Preprint
Jones, C., & Bergen, B. K. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2310.20216 · Cited by 101
Self-citationSymbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95 · cited 1×
“Interrogators’ most frequently provided reasons for their decisions (43”
Does GPT-4 pass the Turing test?
Jones, C., & Bergen, B. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.290 · Cited by 86
Self-citationMoravec, V., Hynek, N., Skare, M., Gavurova, B., & Kubak, M. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.123162 · Cited by 76
Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Models
Ibrahim, L., Akbulut, C., Elasmar, R., Rastogi, C., Kahng, M., Morris, M. R., McKee, K. R., Rieser, V., Shanahan, M., & Weidinger, L. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2502.07077 · Cited by 75
GUNDERSON, K. (1964) · Mind · doi:10.1093/mind/lxxiii.290.234 · Cited by 72
The Cyber-Industrialization of Catfishing and Romance Fraud
Wang, F., & Topalli, V. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.108133 · Cited by 54
Sociotechnical Safety Evaluation of Generative AI Systems
Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., Bergman, S., Kay, J., Griffin, C., Bariach, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.11986 · Cited by 49
Marcus, G., Rossi, F., & Veloso, M. (2016) · AI Magazine · doi:10.1609/aimag.v37i1.2650 · Cited by 48 · cited 5×
“Human interrogators, prone to anthropomorphising, might be fooled by unintelligent systems [16, 27].”
InfluentialCoelho Mollo, D., & Millière, R. (2026) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2026.12307 · Cited by 48
Mollo, D. C., & Millière, R. (2023) · arXiv · doi:10.48550/arxiv.2304.01481 · Cited by 47 · cited 1×
“Interrogators’ most frequently provided reasons for their decisions (43”
Kovács, B. (2024) · Marketing Letters · doi:10.1007/s11002-024-09729-3 · Cited by 44 · cited 1×
“Although a large body of studies have demonstrated that people struggle to discriminate between human and machine generated texts when passively reading them [25, 30, 40], one of the central novel risks of LLMs is their ability to interact dynamically with users [7, 23].”
Human or Not? A Gamified Approach to the Turing Test
Jannai, D., Meron, A., Lenz, B., Levine, Y., & Shoham, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.20010 · Cited by 21
Phishing and Social Engineering in the Age of LLMs
Gallagher, S., Gelman, B., Taoufiq, S., Vörös, T., Lee, Y., Kyadige, A., & Bergeron, S. (2024) · Large Language Models in Cybersecurity · doi:10.1007/978-3-031-54827-7_8 · Cited by 20
Artificial Influence: An Analysis Of AI-Driven Persuasion
Burtell, M., & Woodside, T. (2023) · arXiv · doi:10.48550/arxiv.2303.08721 · Cited by 20
Defending Against Social Engineering Attacks in the Age of LLMs
Ai, L., Kumarage, T. S., Bhattacharjee, A., Liu, Z., Hui, Z., Davinroy, M. S., Cook, J., Cassani, L., Trapeznikov, K., Kirchner, M., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.716 · Cited by 17
Jacobs, O., Pazhoohi, F., & Kingstone, A. (2023) · doi:10.31234/osf.io/pn29d · Cited by 16
Sarkadi, Ş. (2023) · IEEE Technology and Society Magazine · doi:10.1109/mts.2023.3340232 · Cited by 14
Towards interactive evaluations for interaction harms in human-AI systems
Ibrahim, L., Huang, S., Bhatt, U., Ahmad, L., & Anderljung, M. (2024) · arXiv · doi:10.48550/arxiv.2405.10632 · Cited by 11
Towards Interactive Evaluations for Interaction Harms in Human-AI Systems
Ibrahim, L., Huang, S., Ahmad, L., Bhatt, U., & Anderljung, M. (2025) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v8i2.36631 · Cited by 11
A Mechanism-Based Approach to Mitigating Harms from Persuasive Generative AI
El-Sayed, S., Akbulut, C., McCroskery, A., Keeling, G., Kenton, Z., Jalan, Z., Marchal, N., Manzini, A., Shevlane, T., Vallor, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.15058 · Cited by 10
Jones, C. R., & Bergen, B. K. (2024) · arXiv · doi:10.48550/arxiv.2412.17128 · Cited by 7
Self-citationImitation Game: Threshold or Watershed?
Neufeld, E., & Finnestad, S. (2020) · Minds and Machines · doi:10.1007/s11023-020-09544-5 · Cited by 5 · cited 1×
“The most frequent justifications for AI verdicts in Human, GPT-3.”
Experimental evidence that delegating to intelligent machines can increase dishonest behaviour
Köbis, N., Rahwan, Z., Bersch, C., Ajaj, T., Bonnefon, J.-F., & Rahwan, I. (2024) · doi:10.31219/osf.io/dnjgz · Cited by 4
Defending Against Social Engineering Attacks in the Age of LLMsPreprint
Ai, L., Kumarage, T., Bhattacharjee, A., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.12263 · Cited by 1
Werner, T., Soraperra, I., Calvano, E., Parkes, D. C., & Rahwan, I. (2024) · arXiv · doi:10.48550/arxiv.2409.12143 · Cited by 1
The Shadow of Fraud: The Emerging Danger of AI-powered Social Engineering and its Possible Cure
Yu, J., Yu, Y., Wang, X., Lin, Y., Yang, M., Qiao, Y., & Wang, F.-Y. (2024) · arXiv · doi:10.48550/arxiv.2407.15912 · Cited by 1 · cited 1×
“AI systems that can reliably masquerade as people can be used by criminals for fraud [1, 14, 59], by governments for foreign influence [26, 35], and by commercial entities to create engaging but ultimately manipulative experiences for users [7, 57].”
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Large language models and linguistic intentionality
Grindrod, J. (2024) · arXiv · doi:10.48550/arxiv.2404.09576
Research works printed but not linked: 4
- 12. European Parliament and Council. 2024. Regulation (EU) 2024/1183-Artificial Intelligence Act. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L: 2024:206:FULL Official Journal of the European Union L 206/1.
- 17. Patrick Hayes and Kenneth Ford. 1995. Turing Test Considered Harmful. IJCAI 1 (1995), 972-977.
- 37. Graham Oppy and David Dowe. 2021. The Turing Test. In The Stanford En- cyclopedia of Philosophy (winter 2021 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
- 48. Vishvesh Soni. 2023. Large Language Models for Enhancing Customer Lifecycle Management. Journal of Empirical Social Science Studies 7, 1 (Feb. 2023), 67-89.
Cited by
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
From Imitation to Introspection: Probing Self-Consciousness in Language Models
Chen, S., Yu, S., Zhao, S., & Lu, C. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.392 · Cited by 29
There is no such thing as conscious artificial intelligence
Porębski, A., & Figura, J. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05868-8 · Cited by 20
“Indeed, according to the initial fi ndings, some LLMs passed this test (Jones et al., 2025; Jones and Bergen, 2025).”
Larooij, M., & Törnberg, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11412-6 · Cited by 88
“Moreover, generative agents are capable of producing and interpreting natural language in ways that can be indistinguishable from human communication (Jones et al. 2025).”
Duan, J., Li, W., Bai, Q., Nguyen, M., Wang, X., & Jiang, J. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3545822 · Cited by 28
“However, it still falls short of actual humans, who were identified as human 67% of the time [9].”
Metadata
- Authors
- Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K.
- Affiliations
- University of California San Diego (US)
- Venue
- Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2025-06-23
- Pages
- 1615-1639
- OpenAlex
- W4411541869
- Semantic Scholar
- 505c7b471457ac447efdb9abc714247dd10dbef8
- Cited by
- 76
- References
- 36
- Citations by year
- 2026: 38 · 2025: 32
- FWCI
- 66.38
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- University of California, San Diego
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- Turing test, GPT-4, GPT-4o, Eliza, GPT-3.5, human-AI interaction, AI deception, conversational AI, machine intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.