ChatGPT broke the Turing test — the race is on for new ways to assess AI
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“they consistently pass the Turing test (20) as a result (14, 21, 22)”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [14]
In the citing paper
“While a growing number of studies have shown that LLMs exhibit an inherent brittleness (13) and unreliability (14, 15) in knowledge tasks, and that they struggle with real-world understanding (11, 16) and human-like reasoning tasks (9, 10, 14), others have shown that LLMs instead excel in mimicking human language and communicative abilities (17)”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [14]
In the citing paper
“However, this phenomenon changes significantly when the technology already exhibits highly human-like (anthropomorphic) qualities, indistinguishable from the “real deal,” as is evidenced by the Turing Test (14, 21, 22).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [14]
Citations
Cites
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain
Moskvichev, A., Odouard, V. V., & Mitchell, M. (2023) · arXiv · doi:10.48550/arxiv.2305.07141 · Cited by 92
Research works printed but not linked: 6
- Jannai, D., Meron, A., Lenz, B., Levine, Y. & Shoham, Y. Preprint at https://arxiv.org/abs/2305.20010 (2023). · arXiv
- OpenAI. Preprint at https://arxiv.org/abs/2303.08774 (2023). · arXiv
- Chollet, F. Preprint at https://arxiv.org/abs/1911.01547 (2019). · arXiv
- Johnson, A., Vong, W. K., Lake, B. M. & Gureckis, T. M. Preprint at https://arxiv.org/abs/2103.05823 (2021). · arXiv
- Xu, Y., Li, W., Vaezipoor, P., Sanner. S. & Khalil, E. B. Preprint at https://arxiv.org/abs/2305.18354 (2023). · arXiv
- Li, K. et al. Proc. Eleventh Int. Conf. Learn. Represent. https://openreview.net/forum?id=DeG07_TcZvT (2023).
Cited by
Resnik, D. B., & Hosseini, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00493-8 · Cited by 228
“3 Other types of AI discussed in the literature include broad AI (also known as artificial general intelligence or AGI), which is a machine than can perform multiple tasks requiring human-like intelligence; and artificial consciousness (AC), which is a form of AGI with characteristics widely…”
The life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119
“…was put to the test, quite literally, and scored extremely highly on a large variety of standardized tests, in addition to fooling a panel of judges in a version of the Turing test, which led scientists to question the validity of the famous benchmark for machine intelligence (Biever, 2023).”
BackgroundThe benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
Probing out-of-distribution generalization in machine learning for materials
Li, K., Rubungo, A. N., Lei, X., Persaud, D., Choudhary, K. K., DeCost, B., Dieng, A. B., & Hattrick-Simpers, J. R. (2025) · Communications Materials · doi:10.1038/s43246-024-00731-w · Cited by 154
“Similarly, heuristically de fi ned OOD tasks often re fl ect models ’ interpolation capabilities rather than true extrapolation.”
Comparing the Efficacy and Efficiency of Human and Generative AI: Qualitative Thematic Analyses
Prescott, M., Yeager, S., Ham, L., Saldana, C. D. R., Serrano, V. B., Narez, J., Paltin, D., Delgado, J., Moore, D. J., & Montoya, J. L. (2024) · JMIR AI · doi:10.2196/54482 · Cited by 158
“Recently, LLMs have outperformed previously developed artificial intelligence (AI) systems across different tasks spanning a wide range of disciplines [13].”
Background
Metadata
- Authors
- Biever, C.
- Corresponding author
- Biever, Celeste
- Venue
- Nature
- Published
- 2023-07-25
- Volume
- 619
- Issue
- 7971
- Pages
- 686-689
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W4385238793
- Cited by
- 153
- References
- 0
- Citations by year
- 2026: 24 · 2025: 73 · 2024: 42 · 2023: 14
- FWCI
- 13.54
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- ChatGPT, Turing test, large language models, AI evaluation, automated reasoning, human-AI interaction, conversational AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ChatGPT broke the Turing test — the race is on for new ways to assess AI. https://app.cochat.ai/papers/10.1038/d41586-023-02361-7
- The paper
- Biever, C. (2023). ChatGPT broke the Turing test — the race is on for new ways to assess AI. Nature. https://doi.org/10.1038/d41586-023-02361-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.