A Turing test of whether AI chatbots are behaviorally similar to humans
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
We administer a Turing test to AI chatbots. We examine how chatbots behave in a suite of classic behavioral games that are designed to elicit characteristics such as trust, fairness, risk-aversion, cooperation, etc., as well as how they respond to a traditional Big-5 psychological survey that measures personality traits. ChatGPT-4 exhibits behavioral and personality traits that are statistically indistinguishable from a random human from tens of thousands of human subjects from more than 50 countries. Chatbots also modify their behavior based on previous experience and contexts “as if” they were learning from the interactions and change their behavior in response to different framings of the same strategic situation. Their behaviors are often distinct from average and modal human behaviors, in which case they tend to behave on the more altruistic and cooperative end of the distribution. We estimate that they act as if they are maximizing an average of their own and partner’s payoffs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“But users increasingly cannot tell the difference between human writing and LLM writing when presented with text (30–32), and between a human interlocutor or a contemporary chatbot when in interaction (17, 26).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [17]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“research has raised concerns that LLMs may not fully grasp the tasks they perform, sometimes resulting in erratic behavior [54,82].”
From Adaptive Information Modulation: Designing Governance Mechanisms for Multi-Agent Artificial Intelligence Systems (2026) · cites this paper as [54]
Experimental — accuracy not yet measuredIn the citing paper
“Still other studies indicate LLMs can replicate classic experiments in cognitive science and the study of morality and replicate human behavior in the Prisoner’s Dilemma and other behavioral games (21–24).”
From Can Generative AI improve social science? (2024) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
Trust, Reciprocity, and Social History
Berg, J., Dickhaut, J., & McCabe, K. (1995) · Games and Economic Behavior · doi:10.1006/game.1995.1027 · Cited by 5,548
An experimental analysis of ultimatum bargaining
Güth, W., Schmittberger, R., & Schwarze, B. (1982) · Journal of Economic Behavior & Organization · doi:10.1016/0167-2681(82)90011-7 · Cited by 4,816
The structure of phenotypic personality traits.
Goldberg, L. R. (1993) · American Psychologist · doi:10.1037/0003-066x.48.1.26 · Cited by 4,816
Personality trait structure as a human universal.
McCrae, R. R., & Costa, P. T. (1997) · American Psychologist · doi:10.1037/0003-066x.52.5.509 · Cited by 4,234
Theory of Games and Economic Behavior
ROWLAND, E. (1946) · Nature · doi:10.1038/157172a0 · Cited by 3,280
Fairness in Simple Bargaining Experiments
Forsythe, R., Horowitz, J. L., Savin, N. E., & Sefton, M. (1994) · Games and Economic Behavior · doi:10.1006/game.1994.1021 · Cited by 2,366
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
Self-citationRapoport, A., & Chammah, A. (1965) · University of Michigan Press eBooks · doi:10.3998/mpub.20269 · Cited by 1,294
Personality Psychology and Economics
Almlund, M., Duckworth, A. L., Heckman, J., & Kautz, T. (2011) · Handbook of the Economics of Education · doi:10.1016/b978-0-444-53444-6.00001-8 · Cited by 1,204
Cooperation in public-goods experiments: kindness or confusion?
Andreoni, J. (2002) · Market Failure or Success · doi:10.4337/1843760258.00027 · Cited by 912
Back to the future: Personality and Assessment and personality development
Roberts, B. W. (2009) · Journal of Research in Personality · doi:10.1016/j.jrp.2008.12.015 · Cited by 659
The “bomb” risk elicitation task
Crosetto, P., & Filippin, A. (2013) · Journal of Risk and Uncertainty · doi:10.1007/s11166-013-9170-z · Cited by 343
Constant Elasticity of Substitution Production Functions
McFadden, D. (1963) · The Review of Economic Studies · doi:10.2307/2295804 · Cited by 259
Rational Choice and the Framing of Decisions
Tversky, A., & Kahneman, D. (1989) · Multiple Criteria Decision Making and Risk Analysis Using Microcomputers · doi:10.1007/978-3-642-74919-3_4 · Cited by 238
Terwiesch, C., Meincke, L., Girotra, K., Mollick, E., Nave, G., & Ulrich, K. T. (2026) · Production and Operations Management · doi:10.1177/10591478261474243 · Cited by 238
The strategy of conflict . Prospectus for a reorientation of game theory
Schelling, T. C. (1958) · Journal of Conflict Resolution · doi:10.1177/002200275800200301 · Cited by 220
Can GPT-3 pass a writer’s turing test?
Elkins, K., & Chun, J. (2020) · Journal of Cultural Analytics · doi:10.22148/001c.17212 · Cited by 202
The Turing Trap: The Promise & Peril of Human-Like Artificial Intelligence
Brynjolfsson, E. (2022) · Daedalus · doi:10.1162/daed_a_01915 · Cited by 198
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S. M., Frith, C., Ji, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08708 · Cited by 196
AI and Its Impact on Creativity and Diversity: An Empirical Study of LLM-Generated Product Ideas
Terwiesch, C., Meincke, L., Girotra, K., Mollick, E. R., Nave, G., & Ulrich, K. T. (2023) · doi:10.2139/ssrn.4526071 · Cited by 174
The emergence of economic rationality of GPT
Chen, Y., Liu, T. X., Shan, Y., & Zhong, S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2316205120 · Cited by 164
Experienced vs. inexperienced participants in the lab: do they behave differently?
Benndorf, V., Möllers, C., & Normann, H. (2017) · Journal of the Economic Science Association · doi:10.1007/s40881-017-0036-z · Cited by 87
Preplay contracting in the Prisoners’ Dilemma
James Andreoni, A., & Varian, H. R. (1999) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.96.19.10933 · Cited by 84
How do we know how smart AI systems are?
Mitchell, M. (2023) · Science · doi:10.1126/science.adj5957 · Cited by 71
Evidence of general economic principles of bargaining and trade from 2,000 classroom experiments
Lin, P.-H., Brown, A. L., Imai, T., Wang, J. T., Wang, S. W., & Camerer, C. F. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0916-8 · Cited by 50
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
Shapira, N., Levy, M., Alavi, S. H., Zhou, X., Choi, Y., Goldberg, Y., Sap, M., & Shwartz, V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14763 · Cited by 37
Trautman, L. J., Voss, W. G., & Shackelford, S. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4516154 · Cited by 8
A Turing Test: Are AI Chatbots Behaviorally Similar to Humans?Preprint
Mei, Q., Xie, Y., Yuan, W., & Jackson, M. O. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4637354 · Cited by 7
Self-citation
Research works printed but not linked: 1
- K. Warwick, Turing Test Success Marks Milestone in Computing History (University or Reading Press Release, 2014), p. 8.
Cited by
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
Storey, V. C., Yue, W. T., Zhao, J. L., & Lukyanenko, R. (2025) · Information Systems Frontiers · doi:10.1007/s10796-025-10581-7 · Cited by 291
“Similarly, human-sounding outputs of ChatGPT are emergent from the specific connectionist and natural language architecture of GenAI, which, if de-coupled and used separately will fail to render similar types of outputs (Mei et al., 2024).”
A Manager and an AI Walk into a Bar: Does ChatGPT Make Biased Decisions Like We Do?
Chen, Y., Kirshner, S. N., Ovchinnikov, A., Andiappan, M., & Jenkin, T. (2025) · Manufacturing & Service Operations Management · doi:10.1287/msom.2023.0279 · Cited by 143
Metadata
- Authors
- Mei, Q., Xie, Y., Yuan, W., & Jackson, M. O.
- Affiliations
- University of Michigan (US)
- MobLab, Pasadena, CA 91107
- Santa Fe Institute (US)
- Corresponding author
- Mei, Qiaozhu
- Jackson, Matthew O.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2024-02-22
- Volume
- 121
- Issue
- 9
- Pages
- e2313925121-e2313925121
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC10907317
- OpenAlex
- W4392028279
- Cited by
- 254
- References
- 26
- Citations by year
- 2026: 75 · 2025: 118 · 2024: 59
- FWCI
- 104.44
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Experimental Behavioral Economics Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- AI chatbots, Turing test, behavioral games, ChatGPT-4, Big Five personality traits, trust, fairness, risk aversion, cooperation, altruism, experiential learning, framing effects, human-AI comparison
- MeSH terms
- Affect, Altruism, Artificial Intelligence, Behavior, Humans, Pentaerythritol Tetranitrate, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.