The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
A Turing test of whether AI chatbots are behaviorally similar to humans
Open accessAbstract
We administer a Turing test to AI chatbots. We examine how chatbots behave in a suite of classic behavioral games that are designed to elicit characteristics such as trust, fairness, risk-aversion, cooperation, etc., as well as how they respond to a traditional Big-5 psychological survey that measures personality traits. ChatGPT-4 exhibits behavioral and personality traits that are statistically indistinguishable from a random human from tens of thousands of human subjects from more than 50 countries. Chatbots also modify their behavior based on previous experience and contexts “as if” they were learning from the interactions and change their behavior in response to different framings of the same strategic situation. Their behaviors are often distinct from average and modal human behaviors, in which case they tend to behave on the more altruistic and cooperative end of the distribution. We estimate that they act as if they are maximizing an average of their own and partner’s payoffs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“But users increasingly cannot tell the difference between human writing and LLM writing when presented with text (30–32), and between a human interlocutor or a contemporary chatbot when in interaction (17, 26).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [17]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“research has raised concerns that LLMs may not fully grasp the tasks they perform, sometimes resulting in erratic behavior [54,82].”
From Adaptive Information Modulation: Designing Governance Mechanisms for Multi-Agent Artificial Intelligence Systems (2026) · cites this paper as [54]
Experimental — accuracy not yet measuredIn the citing paper
“Still other studies indicate LLMs can replicate classic experiments in cognitive science and the study of morality and replicate human behavior in the Prisoner’s Dilemma and other behavioral games (21–24).”
From Can Generative AI improve social science? (2024) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
Trust, Reciprocity, and Social History
Berg, J., Dickhaut, J., & McCabe, K. (1995) · Games and Economic Behavior · doi:10.1006/game.1995.1027 · Cited by 5,548
An experimental analysis of ultimatum bargaining
Güth, W., Schmittberger, R., & Schwarze, B. (1982) · Journal of Economic Behavior & Organization · doi:10.1016/0167-2681(82)90011-7 · Cited by 4,816
The structure of phenotypic personality traits.
Goldberg, L. R. (1993) · American Psychologist · doi:10.1037/0003-066x.48.1.26 · Cited by 4,816
Personality trait structure as a human universal.
McCrae, R. R., & Costa, P. T. (1997) · American Psychologist · doi:10.1037/0003-066x.52.5.509 · Cited by 4,234
Theory of Games and Economic Behavior
ROWLAND, E. (1946) · Nature · doi:10.1038/157172a0 · Cited by 3,280
Fairness in Simple Bargaining Experiments
Forsythe, R., Horowitz, J. L., Savin, N. E., & Sefton, M. (1994) · Games and Economic Behavior · doi:10.1006/game.1994.1021 · Cited by 2,366
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Research works printed but not linked: 1
- K. Warwick, Turing Test Success Marks Milestone in Computing History (University or Reading Press Release, 2014), p. 8.
Cited by
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
Storey, V. C., Yue, W. T., Zhao, J. L., & Lukyanenko, R. (2025) · Information Systems Frontiers · doi:10.1007/s10796-025-10581-7 · Cited by 289
A Manager and an AI Walk into a Bar: Does ChatGPT Make Biased Decisions Like We Do?
Chen, Y., Kirshner, S. N., Ovchinnikov, A., Andiappan, M., & Jenkin, T. (2025) · Manufacturing & Service Operations Management · doi:10.1287/msom.2023.0279 · Cited by 143
Metadata
- Authors
- Mei, Q., Xie, Y., Yuan, W., & Jackson, M. O.
- Affiliations
- University of Michigan (US)
- MobLab, Pasadena, CA 91107
- Santa Fe Institute (US)
- Corresponding author
- Mei, Qiaozhu
- Jackson, Matthew O.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2024-02-22
- Volume
- 121
- Issue
- 9
- Pages
- e2313925121-e2313925121
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC10907317
- OpenAlex
- W4392028279
- Cited by
- 254
- References
- 26
- Citations by year
- 2026: 75 · 2025: 118 · 2024: 59
- FWCI
- 104.44
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- AI chatbots, Turing test, behavioral games, ChatGPT-4, Big Five personality traits, trust, fairness, risk aversion, cooperation, altruism, experiential learning, framing effects, human-AI comparison
- MeSH terms
- Affect, Altruism, Artificial Intelligence, Behavior, Humans, Pentaerythritol Tetranitrate, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Turing test of whether AI chatbots are behaviorally similar to humans. https://app.cochat.ai/papers/10.1073/pnas.2313925121
- The paper
- Mei, Q., Xie, Y., Yuan, W., & Jackson, M. O. (2024). A Turing test of whether AI chatbots are behaviorally similar to humans. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2313925121
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.