CoChat
Sign Up
Cited by 52Ethics and Social Impacts of AIExperimental Behavioral Economics StudiesComputational and Text Analysis Methods

Take caution in using LLMs as human surrogates

Open access
Gao, Yuan, Lee, Dokyun, Burtch, Gordon, Fazelpour, Sina·2025·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.2501660122PMC PMC12184514
Open this paper in CoChat — free account

Abstract

Recent studies suggest large language models (LLMs) can generate human-like responses, aligning with human behavior in economic experiments, surveys, and political discourse. This has led many to propose that LLMs can be used as surrogates or simulations for humans in social science research. However, LLMs differ fundamentally from humans, relying on probabilistic patterns, absent the embodied experiences or survival objectives that shape human cognition. We assess the reasoning depth of LLMs using the 11-20 money request game. Nearly all advanced approaches fail to replicate human behavior distributions across many models. The causes of failure are diverse and unpredictable, relating to input language, roles, safeguarding, and more. These results warrant caution in using LLMs as surrogates or for simulating human behavior in research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.2501660122

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Assessing the overlap between silicon and human samples is a common practice in current silicon sampling research.

    “instead of assessing the overlap between silicon and human samples, which is a common practice in current silicon sampling research (Gao et al. 2025;Imschloss et al. 2025;Loru et al. 2025;Sarstedt et al. 2024)”

    From Machine marketing: rethinking the customer in the age of generative AI (2026) · cites this paper as Gao et al. 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. [3]J. W. Strachan, Testing theory of mind in large language models and humans. Nat. Hum. Behav. 8, 1285– 1295 (2024). 38769463 PMC11272575
  2. [5]P. Y. Wu, J. Nagler, J. A. Tucker, S. Messing, Large language models can be used to scale the ideologies of politicians in a zero-shot learning setting, April 2023. arXiv [Preprint] (2023). http://arxiv.org/abs/2303.12057 (Accessed 22 January 2025). · arXiv
  3. [6]J. Brand, A. Israeli, D. Ngwe, Using GPT for market research (2023). 10.2139/ssrn.4395751. Accessed 22 January 2025. · DOI · Paper page
  4. [8]H. Jiang, X. Zhang, X. Cao, J. Kabbara, Personallm: Investigating the ability of large language models to express big five personality traits. arXiv [Preprint] (2023). http://arxiv.org/abs/2305.02547 (Accessed 22 January 2025). · arXiv
  5. [9]G. V. Aher, R. I. Arriaga, A. T. Kalai, “Using large language models to simulate multiple humans and replicate human subject studies” in International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, J. Scarlett, Eds. (PMLR, 2023), pp. 337–371.
  6. [15]R. Burnell, Rethink reporting of evaluation results in AI. Science 380, 136– 138 (2023). 37053341
  7. [16]J. Burden, Evaluating AI evaluation: Perils and prospects. arXiv [Preprint] (2024). http://arxiv.org/abs/2407.09221 (Accessed 22 January 2025). · arXiv
  8. [18]J. Kaplan et al., Scaling laws for neural language models. arXiv [Preprint] (2020). http://arxiv.org/abs/2001.08361 (Accessed 22 January 2025). · arXiv
  9. [19]M. Binz et al., Centaur: A foundation model of human cognition. arXiv [Preprint] (2024). https://arxiv.org/abs/2410.20268 (Accessed 22 January 2025). · arXiv
  10. [20]S. Lu, Strategic interactions between large language models-based agents in beauty contests. arXiv [Preprint] (2024). http://arxiv.org/abs/2404.08492 (Accessed 22 January 2025). · arXiv
  11. [21]S. Guo et al., Economics arena for large language models. arXiv [Preprint] (2024). http://arxiv.org/abs/2401.01735 (Accessed 22 January 2025). · arXiv
  12. [22]D. Dagaev, S. Paklina, P. Parshakov, Strategizing with AI: Insights from a beauty contest Experiment (2024). 10.2139/ssrn.4754435. Accessed 22 January 2025. · DOI
  13. [27]W. Street et al., LLMs achieve adult human performance on higher-order theory of mind tasks. arXiv [Preprint] (2024). http://arxiv.org/abs/2405.18870 (Accessed 22 January 2025). PMC12808479 41551539 · arXiv
  14. [28]L. Zhou, Larger and more instructable language models become less reliable. Nature 634, 61– 68 (2024). 39322679 PMC11446866
  15. [29]T. B. Brown et al., Language models are few-shot learners. arXiv [Preprint] (2020). 10.48550/arXiv.2005.14165. · DOI · Paper page
  16. [30]P. Lewis, Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv. Neural Inf. Process. Syst. 33, 9459– 9474 (2020).
  17. [31]J. Howard, S. Ruder, Universal language model fine-tuning for text classification. arXiv [Preprint] (2018). http://arxiv.org/abs/1801.06146 (Accessed 22 January 2025). · arXiv
  18. [32]J. Wei, Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. Syst. 35, 24824– 24837 (2022).
  19. [34]C. Li et al., Large language models understand and can be enhanced by emotional stimuli. arXiv [Preprint] (2023). http://arxiv.org/abs/2307.11760 (Accessed 22 January 2025). · arXiv
  20. [35]M. Hutson, How does chatGPT "think"? Psychology and neuroscience crack open AI large language models Nature 629, 986– 988 (2024). 38745024
  21. [36]G. Gui, O. Toubia, The challenge of using LLMs to simulate human behavior: A causal inference perspective. arXiv [Preprint] (2023). http://arxiv.org/abs/2312.15524 (Accessed 22 January 2025). · arXiv
  22. [37]X. Jiang et al., A survey on large language model hallucination via a creativity perspective. arXiv [Preprint] (2024). http://arxiv.org/abs/2402.06647 (Accessed 22 January 2025). · arXiv
  23. [38]M. Lee, A mathematical investigation of hallucination and creativity in GPT models. Mathematics 11, 2320 (2023).
  24. [40]T. Hagendorff, Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. arXiv [Preprint] (2023). http://arxiv.org/abs/2303.13988 (Accessed 22 January 2025). · arXiv
  25. [43]K. Matton, R. Ness, J. Guttag, E. Kiciman, “Walk the talk? Measuring the faithfulness of large language model explanations” in The Thirteenth International Conference on Learning Representations (OpenReview.net, 2025).
  26. [44]M. Turpin, J. Michael, E. Perez, S. Bowman, Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Adv. Neural Inf. Process. Syst. 36, 74952–74965 (2023).
  27. [45]X. Ye, G. Durrett, The unreliability of explanations in few-shot prompting for textual reasoning. Adv. Neural Inf. Process. Syst. 35, 30378– 30392 (2022).
  28. [46]I. Mirzadeh et al., GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv [Preprint] (2024). http://arxiv.org/abs/2410.05229. · arXiv
  29. [47]M. A. Boden, The Philosophy of Artificial Intelligence (Oxford University Press, Inc., 1990).
  30. [48]Y. Dong, X. Jiang, H. Liu, Z. Jin, G. Li, Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv [Preprint] (2024). http://arxiv.org/abs/2402.15938 (Accessed 22 January 2025). · arXiv
  31. [50]M. Nezhurina, L. Cipolina-Kun, M. Cherti, J. Jitsev, Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv [Preprint] (2024). http://arxiv.org/abs/2406.02061 (Accessed 22 January 2025). · arXiv
  32. [51]S. Williams, J. Huckle, Easy problems that LLMs get wrong. arXiv [Preprint] (2024). http://arxiv.org/abs/2405.19616 (Accessed 22 January 2025). · arXiv
  33. [52]Q. Bao et al., A systematic evaluation of large language models on out-of-distribution logical reasoning tasks. arXiv [Preprint] (2023). http://arxiv.org/abs/2310.09430 (Accessed 22 January 2025). · arXiv
  34. [53]M. Parmar et al., Towards systematic evaluation of logical reasoning ability of large language models. arXiv [Preprint] (2024). http://arxiv.org/abs/2404.15522 (Accessed 22 January 2025). · arXiv
  35. [54]Z. Jin et al., Can large language models infer causation from correlation? arXiv [Preprint] (2023). http://arxiv.org/abs/2306.05836 (Accessed 22 January 2025). · arXiv
  36. [55]C. Yuan, Q. Xie, J. Huang, S. Ananiadou, “Back to the future: Towards explainable temporal reasoning with large language models” in Proceedings of the ACM on Web Conference 2024 (Association for Computing Machinery (ACM), 2024), pp. 1963–1974.
  37. [56]P. Laban et al., LLMs as factual reasoners: Insights from existing benchmarks and beyond. arXiv [Preprint] (2023). http://arxiv.org/abs/2305.14540 (Accessed 22 January 2025). · arXiv
  38. [57]A. Saparov, Testing the general deductive reasoning capacity of large language models using OOD examples. Adv. Neural Inf. Process. Syst. 36, 3083– 3105 (2024).
  39. [58]G. Gendron, Q. Bao, M. Witbrock, G. Dobbie, Large language models are not strong abstract reasoners. arXiv [Preprint] (2023). http://arxiv.org/abs/2305.19555 (Accessed 22 January 2025). · arXiv
  40. [59]L. Berglund et al., The reversal curse: LLMs trained on “a is b” fail to learn “b is a”. arXiv [Preprint] (2023). http://arxiv.org/abs/2309.12288 (Accessed 22 January 2025). · arXiv
  41. [60]Z. Yang et al., Can large language models always solve easy problems if they can solve harder ones? arXiv [Preprint] (2024). http://arxiv.org/abs/2406.12809 (Accessed 22 January 2025). · arXiv
  42. [61]J. Jiang, F. Wang, J. Shen, S. Kim, S. Kim, A survey on large language models for code generation. arXiv [Preprint] (2024). http://arxiv.org/abs/2406.00515 (Accessed 22 January 2025). · arXiv
  43. [62]C. E. Jimenez et al., Can language models resolve real-world Github issues? arXiv [Preprint] (2023). http://arxiv.org/abs/2310.06770 (Accessed 22 January 2025). · arXiv
  44. [63]T. Ullman, Large language models fail on trivial alterations to theory-of-mind tasks. arXiv [Preprint] (2023). http://arxiv.org/abs/2302.08399. · arXiv
  45. [64]J. Pfau, W. Merrill, S. R. Bowman, Let’s think dot by dot: Hidden computation in transformer language models. arXiv [Preprint] (2024). http://arxiv.org/abs/2404.15758 (Accessed 22 January 2025). · arXiv
  46. [65]S. Min et al., Rethinking the role of demonstrations: What makes in-context learning work? arXiv [Preprint] (2022). https://arxiv.org/abs/2202.12837 (Accessed 22 January 2025). · arXiv
  47. [66]M. Sclar, Y. Choi, Y. Tsvetkov, A. Suhr, Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv [Preprint] (2023). http://arxiv.org/abs/2310.11324 (Accessed 22 January 2025). · arXiv
  48. [67]T. Ceron, N. Falk, A. Barić, D. Nikolaev, S. Padó, Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs. arXiv [Preprint] (2024). http://arxiv.org/abs/2402.17649. · arXiv
  49. [68]V. Gupta, D. Pantoja, C. Ross, A. Williams, M. Ung, Changing answer order can decrease MMLU accuracy. arXiv [Preprint] (2024). http://arxiv.org/abs/2406.19470 (Accessed 22 January 2025). · arXiv
  50. [69]N. Carlini et al., “Extracting training data from large language models” in 30th USENIX Security Symposium (USENIX Security 21), M. Bailey, R. Greenstadt, Eds. (USENIX Association, 2021), pp. 2633–2650.
  51. [71]Z. Wu et al., Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. arXiv [Preprint] (2023). http://arxiv.org/abs/2307.02477 (Accessed 22 January 2025). · arXiv
  52. [72]M. Lewis, M. Mitchell, Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiv [Preprint] (2024). https://arxiv.org/abs/2402.08955 (Accessed 22 January 2025). · arXiv
  53. [73]K. Zhou et al., Don’t make your LLM an evaluation benchmark cheater. arXiv [Preprint] (2023). http://arxiv.org/abs/2311.01964 (Accessed 22 January 2025). · arXiv
  54. [74]C. Deng et al., Unveiling the spectrum of data contamination in language models: A survey from detection to remediation. arXiv [Preprint] (2024). http://arxiv.org/abs/2406.14644 (Accessed 22 January 2025). · arXiv
  55. [76]E. Zhou, D. Lee, Generative artificial intelligence, human creativity, and art. PNAS Nexus 3, pgae052 (2024). 38444602 PMC10914360
  56. [77]C. D. Whitney, J. Norman, “Real risks of fake data: Synthetic data, diversity-washing and consent circumvention” in The 2024 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery (ACM), 2024), pp. 1733–1744.
  57. [78]W. Agnew et al., “The illusion of artificial inclusion” in Proceedings of the CHI Conference on Human Factors in Computing Systems (Association for Computing Machinery (ACM), 2024), pp. 1–12.

Cited by

Sorted by citations

Authors
Gao, Y., Lee, D., Burtch, G., & Fazelpour, S.
Affiliations
  • Boston University (US)
  • Northeastern University (US)
Corresponding author
  • Gao, Yuan
  • Lee, Dokyun
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2025-06-13
Volume
122
Issue
24
Pages
e2501660122-e2501660122
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.2501660122
PMC
PMC12184514
OpenAlex
W4411272807
Cited by
52
References
9
Citations by year
2026: 36 · 2025: 16
FWCI
53.81
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Safety Research › Experimental Behavioral Economics Studies
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
large language models, human behavior simulation, reasoning depth, embodied cognition, social science research
MeSH terms
Cognition, Humans, Language

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review