CoChat
Sign Up
PreprintCited by 161Topic ModelingEthics and Social Impacts of AINatural Language Processing Techniques

Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

Open access
Aher, Gati, Arriaga, Rosa I., Kalai, Adam Tauman·2022·arXiv
DOI 10.48550/arxiv.2208.10264arXiv 2208.10264
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a new type of test, called a Turing Experiment (TE), for evaluating to what extent a given language model, such as GPT models, can simulate different aspects of human behavior. A TE can also reveal consistent distortions in a language model's simulation of a specific human behavior. Unlike the Turing Test, which involves simulating a single arbitrary individual, a TE requires simulating a representative sample of participants in human subject research. We carry out TEs that attempt to replicate well-established findings from prior studies. We design a methodology for simulating TEs and illustrate its use to compare how well different language models are able to reproduce classic economic, psycholinguistic, and social psychology experiments: Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds. In the first three TEs, the existing findings were replicated using recent models, while the last TE reveals a "hyper-accuracy distortion" present in some language models (including ChatGPT and GPT-4), which could affect downstream applications in education and the arts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2208.10264

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing studies remain narrowly focused on isolated problems and rely on simplified environments, such as text-based or simplistic game environments, with limited attention to real-world environment fidelity.

    “Existing studies remain narrowly focused on isolated problems and rely on simplified environments, such as text-based [4] or simplistic game environments [80], with limited attention to real-world environment fidelity, revealing notable limitations.”

    From AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society (2025) · cites this paper as [4]

  • LLM-based agents have successfully reproduced experimental results in psychology and linguistics, making them valuable tools for in silico social experiments.

    “Despite these biases, LLM-based agents have successfully reproduced experimental results in psychology and linguistics [41], making them valuable tools for in silico social experiments.”

    From Selective agreement, not sycophancy: investigating opinion dynamics in LLM interactions (2025) · cites this paper as [41]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 47

  1. Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., and Wingate, D. Out of one, many: Using language models to simulate human samples. Political Analysis , pp. 1-15, 2023. doi: 10.1017/pan.2023.2. · DOI · Paper page
  2. Bakker, M., Chadwick, M., Sheahan, H., Tessler, M., Campbell-Gillingham, L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M., et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems , 35:38176-38189, 2022.
  3. Bender, E. M. and Koller, A. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics , pp. 5185-5198, 2020.
  4. Binz, M. and Schulz, E. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences , 120(6):e2218523120, 2023.
  5. Blodgett, S. L., Barocas, S., Daum´ e III, H., and Wallach, H. Language (technology) is power: A critical survey of 'bias' in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 5454-5476, 2020.
  6. Bolton, J. M., Gunnell, D., and Turecki, G. Suicide risk assessment and intervention in people with mental illness. BMJ , 351, 2015. doi: 10.1136/bmj.h4978. URL https://www.bmj.com/content/351/bmj.h4978 . · DOI · Paper page
  7. Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., and Kalai, A. T. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems (NeurIPS) , 2016.
  8. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: //proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .
  9. Caron, G. and Srivastava, S. Identifying and manipulating the personality traits of language models. arXiv preprint arXiv:2212.10276 , 2022. · arXiv
  10. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. URL https://arxiv.org/abs/2204.02311 . · arXiv
  11. Christianson, K., Hollingworth, A., Halliwell, J. F., and Ferreira, F. Thematic roles assigned along the garden path linger. Cognitive psychology , 42(4):368-407, 2001.
  12. Crain, S. and Steedman, M. On not being led up the garden path: the use of context by the psychological syntax processor , pp. 320-358. Cambridge University Press, United States, 1985. ISBN 9780521262033. Cambridge Books Online.
  13. Dasgupta, I., Lampinen, A. K., Chan, S. C., Creswell, A., Kumaran, D., McClelland, J. L., and Hill, F. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051 , 2022. · arXiv
  14. Eckel, C. C. and Grossman, P. J. Chivalry and Solidarity in Ultimatum Games. Economic Inquiry , 39(2): 171-188, April 2001. URL https://ideas.repec.org/a/oup/ecinqu/v39y2001i2p171-88.html .
  15. Galton, F. Vox populi. Nature , 75(7):450-451, 1907.
  16. G¨ uth, W., Schmittberger, R., and Schwarze, B. An experimental analysis of ultimatum bargaining. Journal of Economic Behavior & Organization , 3(4):367-388, 1982. ISSN 0167-2681. doi: https: //doi.org/10.1016/0167-2681(82)90011-7. URL https://www.sciencedirect.com/science/article/ pii/0167268182900117 . · DOI
  17. Hagendorff, T., Fabi, S., and Kosinski, M. Machine intuition: Uncovering human-like intuitive decision-making in gpt-3.5. arXiv preprint arXiv:2212.05206 , 2022. · arXiv
  18. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR) , 2021.
  19. Houser, D. and McCabe, K. Chapter 2 - experimental economics and experimental game theory. In Glimcher, P. W. and Fehr, E. (eds.), Neuroeconomics (Second Edition) , pp. 19-34. Academic Press, San Diego, second edition edition, 2014. ISBN 978-0-12-416008-8. doi: https://doi.org/10.1016/B978-0-12-416008-8.00002-4. URL https://www.sciencedirect.com/science/article/pii/B9780124160088000024 . · DOI
  20. Jiang, G., Xu, M., Zhu, S.-C., Han, W., Zhang, C., and Zhu, Y. Mpi: Evaluating and inducing personality in pre-trained language models. arXiv preprint arXiv:2206.07550 , 2022. · arXiv
  21. Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. L., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., and Choi, Y. Delphi: Towards machine ethics and norms. ArXiv , abs/2110.07574, 2021. URL https://arxiv.org/abs/2110.07574 . · arXiv
  22. Jones, E. and Steinhardt, J. Capturing failures of large language models via human cognitive biases. arXiv preprint arXiv:2202.12299 , 2022. · arXiv
  23. Karra, S. R., Nguyen, S., and Tulabandhula, T. Ai personification: Estimating the personality of language models. arXiv preprint arXiv:2204.12000 , 2022. · arXiv
  24. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners, 2022. URL https://arxiv.org/abs/2205.11916 . · arXiv
  25. Kosinski, M. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083 , 2023. · arXiv
  26. Krawczyk, D. C. Chapter 12 - social cognition: Reasoning with others. In Krawczyk, D. C. (ed.), Reasoning , pp. 283-311. Academic Press, 2018. ISBN 978-0-12-809285-9. doi: https://doi.org/10.1016/B978-0-12-809285-9. 00012-0. URL https://www.sciencedirect.com/science/article/pii/B9780128092859000120 . · DOI
  27. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., R´ e, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wang, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N., Khattab, O., Henderson, P., Huang, Q., Chi, R., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. Holistic evaluation of language models, 2022. URL https://arxiv.org/abs/2211.09110 . · arXiv
  28. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021. URL https://arxiv.org/abs/2107. 13586 .
  29. Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. CoRR , abs/2104.08786, 2021. URL https: //arxiv.org/abs/2104.08786 . · arXiv
  30. Macal, C. and North, M. Tutorial on agent-based modelling and simulation. Journal of Simulation , 4(3): 151-162, 2010.
  31. Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics , 19(2):313-330, June 1993.
  32. Messias, J., Vikatos, P., and Benevenuto, F. White, man, and highly followed: Gender and race inequalities in twitter. In Proceedings of the International Conference on Web Intelligence , pp. 266-274, 2017.
  33. Milgram, S. Behavioral study of obedience. The Journal of abnormal and social psychology , 67(4):371, 1963.
  34. Moussa¨ ıd, M., K¨ ammer, J. E., Analytis, P. P., and Neth, H. Social influence and the collective dynamics of opinion formation. PLOS ONE , 8(11):1-8, 11 2013. doi: 10.1371/journal.pone.0078433. URL https: //doi.org/10.1371/journal.pone.0078433 . · DOI · Paper page
  35. OpenAI. GPT-4 Technical Report, March 2023. URL http://arxiv.org/abs/2303.08774 . arXiv:2303.08774 [cs]. · arXiv
  36. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.
  37. Page, S. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. Princeton University Press, 2007.
  38. Park, J. S., Popowski, L., Cai, C., Morris, M. R., Liang, P., and Bernstein, M. S. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology , pp. 1-18, 2022.
  39. Patson, N. D., Darowski, E. S., Moon, N., and Ferreira, F. Lingering misinterpretations in garden-path sentences: evidence from a paraphrasing task. Journal of experimental psychology. Learning, memory, and cognition , 35 1:280-5, 2009.
  40. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 42754293, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.330. URL https://aclanthology.org/2021.acl-long.330 . · DOI · Paper page
  41. Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 4222-4235, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.346. URL https: //aclanthology.org/2020.emnlp-main.346 . · DOI · Paper page
  42. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., and (422-others). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022. URL https://arxiv.org/abs/2206.04615 . · arXiv
  43. Ullman, T. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399 , 2023. · arXiv
  44. Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating gender bias in language models using causal mediation analysis. Advances in Neural Information Processing Systems , 33:12388-12401, 2020.
  45. Warwick, K. and Shah, H. Can machines think? a report on turing test experiments at the royal society. Journal of experimental & Theoretical artificial Intelligence , 28(6):989-1007, 2016.
  46. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021. · arXiv
  47. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models, 2022. URL https://arxiv.org/abs/2201. 11903 .

Cited by

Sorted by citations

Authors
Aher, G., Arriaga, R. I., & Kalai, A. T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2208.10264
arXiv
2208.10264
OpenAlex
W4292947474
Cited by
161
References
0
Citations by year
2026: 9 · 2025: 59 · 2024: 58 · 2023: 34 · 2022: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, human behavior simulation, GPT-4, ChatGPT, ultimatum game, garden-path sentences, wisdom of crowds, psycholinguistics, social psychology experiments, economic experiments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review