The DOI matches its DataCite record.
Human or Not? A Gamified Approach to the Turing Test
Open accessAbstract
We present "Human or Not?", an online game inspired by the Turing test, that measures the capability of AI chatbots to mimic humans in dialog, and of humans to tell bots from other humans. Over the course of a month, the game was played by over 1.5 million users who engaged in anonymous two-minute chat sessions with either another human or an AI language model which was prompted to behave like humans. The task of the players was to correctly guess whether they spoke to a person or to an AI. This largest scale Turing-style test conducted to date revealed some interesting facts. For example, overall users guessed the identity of their partners correctly in only 68% of the games. In the subset of the games in which users faced an AI bot, users had even lower correct guess rates of 60% (that is, not much higher than chance). This white paper details the development, deployment, and results of this unique experiment. While this experiment calls for many extensions and refinements, these findings already begin to shed light on the inevitable near future which will commingle humans and AI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A recent large-scale study found humans were 60% accurate in identifying modern language models in two-minute online conversations.
“A recent large-scale study [21] found that humans were 60% accurate in identifying a range of modern language models in two-minute online conversations.”
From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [21]
The human pass rate is similar to other recent estimates.
“and is similar to other recent estimates [21].”
From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 3
- OpenAI. GPT-4 Technical Report, 2023. URL https://arxiv.org/pdf/2303.08774. pdf. · arXiv
- Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-Context Retrieval-Augmented Language Models. arXiv preprint arXiv:2302.00083, 2023. · arXiv
- Alan M. Turing. Computing Machinery and Intelligence. Communications of the ACM, 59:433-460, 1950.
Cited by
Sorted by citations
People cannot distinguish GPT-4 from a human in a Turing test
Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732108 · Cited by 76
Ten simple rules to leverage large language models for getting grants
Seckel, E., Stephens, B. Y., & Rodriguez, F. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011863 · Cited by 14
Metadata
- Authors
- Jannai, D., Meron, A., Lenz, B., Levine, Y., & Shoham, Y.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.20010
- OpenAlex
- W4379089685
- Cited by
- 21
- References
- 0
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 6 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- Turing test, AI chatbots, gamification, online games, large language models, human-AI interaction, human-AI coexistence, deception detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
