CoChat
Sign Up
PreprintCited by 3Evolutionary Game Theory and CooperationEthics and Social Impacts of AIAccess Control and Trust

Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents

Open access
Buscemi, Alessio, Proverbio, Daniele, Bova, Paolo, Balabanova, Nataliya, Bashir, Adeela, Cimpeanu, Theodor, da Fonseca, Henrique Correia, Duong, Manh Hong +10 more·2025·arXiv
DOI 10.48550/arxiv.2504.08640arXiv 2504.08640
Open this paper in CoChat — free accountOpen PDF

Abstract

There is general agreement that fostering trust and cooperation within the AI development ecosystem is essential to promote the adoption of trustworthy AI systems. By embedding Large Language Model (LLM) agents within an evolutionary game-theoretic framework, this paper investigates the complex interplay between AI developers, regulators and users, modelling their strategic choices under different regulatory scenarios. Evolutionary game theory (EGT) is used to quantitatively model the dilemmas faced by each actor, and LLMs provide additional degrees of complexity and nuances and enable repeated games and incorporation of personality traits. Our research identifies emerging behaviours of strategic AI agents, which tend to adopt more "pessimistic" (not trusting and defective) stances than pure game-theoretic agents. We observe that, in case of full trust by users, incentives are effective to promote effective regulation; however, conditional trust may deteriorate the "social pact". Establishing a virtuous feedback between users' trust and regulators' reputation thus appears to be key to nudge developers towards creating safe AI. However, the level at which this trust emerges may depend on the specific LLM used for testing. Our results thus provide guidance for AI regulation systems, and help predict the outcome of strategic LLM agents, should they be used to aid regulation itself.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.08640

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current research on agentic systems has focused mainly on individual agent performance and structured interactions between agents for isolated economic tasks.

    “Current research on agentic systems has focused mainly on individual agent performance and structured interactions between agents for isolated economic tasks (Wang et al., 2024; Buscemi et al., 2025; Mao et al., 2025; Abdelnabi et al., 2024).”

    From Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets (2025) · cites this paper as Buscemi et al., 2025

    The Introduction (p. 2) describes structured interactions among three LLM agents in a game-theoretic regulatory setting, so the paper supports that part of the sentence. It does not establish that current agentic-systems research focuses mainly on individual agent performance or isolated economic tasks; its own task models a regulatory ecosystem. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 51

  1. [1]Y. Bengio, G. Hinton, A. Yao, D. Song, P. Abbeel et al. , 'Managing extreme AI risks amid rapid progress,' Science , vol. 384, no. 6698, pp. 842-845, May 2024. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.adn0117 · DOI · Paper page
  2. [2]L. Floridi, J. Cowls, M. Beltrametti, R. Chatila, P. Chazerand, V. Dignum, C. Luetge, R. Madelin, U. Pagallo, F. Rossi et al. , 'Ai4people-an ethical framework for a good ai society: opportunities, risks, principles, and recommendations,' Minds and machines , vol. 28, pp. 689-707, 2018.
  3. [4]G. Finocchiaro, 'The regulation of artificial intelligence,' AI & SOCIETY , vol. 39, no. 4, pp. 1961-1968, 2024.
  4. [5]L. Hammond, A. Chan, J. Clifton, J. Hoelscher-Obermaier, A. Khan, E. McLean, C. Smith, W. Barfuss, J. Foerster, T. Gavenˇ ciak, T. A. Han, E. Hughes, V. Kovaˇ r´ ık, J. Kulveit, J. Z. Leibo, C. Oesterheld, C. S. de Witt, N. Shah, M. Wellman, P. Bova, T. Cimpeanu, C. Ezell, Q. Feuillade-Montixi, M. Franklin, E. Kran, I. Krawczuk, M. Lamparth, N. Lauffer, A. Meinke, S. Motwani, A. Reuel, V. Conitzer, M. Dennis, I. Gabriel, A. Gleave, G. Hadfield, N. Haghtalab, A. Kasirzadeh, S. Krier, K. Larson, J. Lehman, D. C. Parkes, G. Piliouras, and I. Rahwan, 'Multi-agent risks from advanced ai,' 2025. [Online]. Available: https://arxiv.org/abs/2502.14143 · arXiv
  5. [6]Y. Bengio, S. Mindermann, D. Privitera, T. Besiroglu, R. Bommasani, S. Casper, Y. Choi, P. Fox, B. Garfinkel, D. Goldfarb et al. , 'International ai safety report,' arXiv preprint arXiv:2501.17805 , 2025. · arXiv
  6. [7]S. T. Powers, O. Linnyk, M. Guckert, J. Hannig, J. Pitt, N. Urquhart, A. Ek´ art, N. Gumpfer, T. A. Han, P. R. Lewis et al. , 'The stuff we swim in: regulation alone will not lead to justifiable trust in ai,' IEEE Technology and Society Magazine , vol. 42, no. 4, pp. 95-106, 2024.
  7. [8]J. Laux, S. Wachter et al. , 'Trustworthy artificial intelligence and the European Union AI act: On the conflation of trustworthiness and acceptability of risk,' Regulation & Governance , vol. 18, no. 1, pp. 3-32, 2024.
  8. [10]J. Tallberg, E. Erman et al. , 'The global governance of artificial intelligence: Next steps for empirical and normative research,' International Studies Review , vol. 25, no. 3, p. viad040, 2023, private vs public regulation.
  9. [11]J. Clark and G. K. Hadfield, 'Regulatory Markets for AI Safety,' arXiv , Dec. 2019.
  10. [13]R. Clarke, 'Regulatory alternatives for ai,' Computer Law & Security Review , vol. 35, no. 4, pp. 398-409, 2019.
  11. [14]A. Dafoe, 'AI Governance: Overview and Theoretical Lenses,' in The Oxford Handbook of AI Governance , J. B. Bullock, Y.-C. Chen, J. Himmelreich, V. M. Hudson, A. Korinek, M. M. Young, and B. Zhang, Eds. Oxford University Press, 2023, p. 0.
  12. [16]E. Zaidan and I. A. Ibrahim, 'Ai governance in a complex and rapidly changing regulatory landscape: A global perspective,' Humanities and Social Sciences Communications , vol. 11, no. 1, pp. 1-18, 2024.
  13. [17]Z. Alalawi, P. Bova, T. Cimpeanu, A. Di Stefano, M. H. Duong, E. F. Domingos, T. A. Han, M. Krellner, B. Ogbo, S. T. Powers et al. , 'Trust ai regulation? discerning users are vital to build trust and effective ai regulation,' arXiv preprint arXiv:2403.09510 , 2024. · arXiv
  14. [18]D. Kondor, V. Hafez, S. Shankar, R. Wazir, and F. Karimi, 'Complex systems perspective in assessing risks in artificial intelligence,' Philosophical Transactions A , vol. 382, no. 2285, p. 20240109, 2024.
  15. [19]K. J. D. Chan, G. Papyshev, and M. Yarime, 'Balancing the tradeoff between regulation and innovation for artificial intelligence: An analysis of top-down command and control and bottom-up self-regulatory approaches,' Technology in Society , vol. 79, p. 102747, 2024.
  16. [20]P. Bova, A. Di Stefano, and T. A. Han, 'Both eyes open: Vigilant incentives help auditors improve ai safety,' Journal of Physics: Complexity , vol. 5, no. 2, p. 025009, 2024.
  17. [21]T. A. Han, L. M. Pereira et al. , ' To Regulate or Not: A Social Dynamics Analysis of an Idealised AI Race ,' Journal of Artificial Intelligence Research , vol. 69, pp. 881-921, Nov. 2020.
  18. [24]T. A. Han, C. Perret, and S. T. Powers, 'When to (or not to) trust intelligent machines: Insights from an evolutionary game theory analysis of trust in repeated games,' Cognitive Systems Research , vol. 68, pp. 111-124, 2021.
  19. [25]A. Buscemi and D. Proverbio, 'Roguegpt: dis-ethical tuning transforms chatgpt4 into a rogue ai in 158 words,' arXiv preprint arXiv:2407.15009 , 2024. · arXiv
  20. [26]P. Andras, L. Esterle, M. Guckert, T. A. Han, P. R. Lewis, K. Milanovic, T. Payne, C. Perret, J. Pitt, S. T. Powers et al. , 'Trusting intelligent machines: Deepening trust within socio-technical systems,' IEEE Technology and Society Magazine , vol. 37, no. 4, pp. 76-83, 2018.
  21. [27]S. Armstrong, N. Bostrom et al. , ' Racing to the Precipice: A Model of Artificial Intelligence Development ,' Ai & Society , vol. 31, no. 2, pp. 201-206, May 2016.
  22. [28]A. Askell, M. Brundage et al. , 'The Role of Cooperation in Responsible AI Development,' arXiv , Jul. 2019.
  23. [30]P. Cihon, M. J. Kleinaltenkamp et al. , ' AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries ,' IEEE Transactions on Technology and Society , vol. 2, no. 4, pp. 200-209, Dec. 2021.
  24. [31]D. C. North, Institutions, institutional change and economic performance . Cambridge university press, 1990.
  25. [32]J. Hofbauer and K. Sigmund, Evolutionary games and population dynamics . Cambridge university press, 1998.
  26. [33]K. Sigmund, 'The calculus of selfishness,' in The Calculus of Selfishness . Princeton University Press, 2010.
  27. [34]J. S. Park, J. O'Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, 'Generative agents: Interactive simulacra of human behavior,' in Proceedings of the 36th annual acm symposium on user interface software and technology , 2023, pp. 1-22.
  28. [35]C. A. Bail, 'Can generative ai improve social science?' Proceedings of the National Academy of Sciences , vol. 121, no. 21, p. e2314021121, 2024.
  29. [36]W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al. , 'A survey of large language models,' arXiv preprint arXiv:2303.18223 , vol. 1, no. 2, 2023. · arXiv
  30. [37]A. Buscemi and D. Proverbio, 'Large language models' detection of political orientation in newspapers,' arXiv preprint arXiv:2406.00018 , 2024. · arXiv
  31. [38]--, 'Chatgpt vs gemini vs llama on multilingual sentiment analysis,' arXiv preprint arXiv:2402.01715 , 2024. · arXiv
  32. [39]N. Lee, J. Hong, and J. Thorne, 'Evaluating the consistency of llm evaluators,' arXiv preprint arXiv:2412.00543 , 2024. · arXiv
  33. [43]Z. Song and T. A. Han, 'On evolution of non-binding commitments,' Physics of Life Reviews , vol. 52, pp. 245-247, 2025.
  34. [44]P. A. Van Lange, J. Joireman, C. D. Parks, and E. Van Dijk, 'The psychology of social dilemmas: A review,' Organizational Behavior and Human Decision Processes , vol. 120, no. 2, pp. 125-141, 2013.
  35. [45]N. Balabanova, A. Bashir, P. Bova, A. Buscemi, T. Cimpeanu, H. C. da Fonseca, A. Di Stefano, M. H. Duong, E. F. Domingos, A. Fernandes et al. , 'Media and responsible ai governance: a game-theoretic and llm analysis,' arXiv preprint arXiv:2503.09858 , 2025. · arXiv
  36. [46]M. A. Nowak, 'Five rules for the evolution of cooperation,' science , vol. 314, no. 5805, pp. 1560-1563, 2006.
  37. [47]S. Van Segbroeck, J. M. Pacheco, T. Lenaerts, and F. C. Santos, 'Emergence of fairness in repeated group interactions,' Physical review letters , vol. 108, no. 15, p. 158104, 2012.
  38. [49]M. A. Nowak, A. Sasaki, C. Taylor, and D. Fudenberg, 'Emergence of cooperation and evolutionary stability in finite populations,' Nature , vol. 428, no. 6983, pp. 646-650, 2004.
  39. [50]I. Zisis, S. Di Guida, T. A. Han, G. Kirchsteiger, and T. Lenaerts, 'Generosity motivated by acceptanceevolutionary analysis of an anticipation game,' Scientific reports , vol. 5, no. 1, p. 18076, 2015.
  40. [51]D. G. Rand, C. E. Tarnita, H. Ohtsuki, and M. A. Nowak, 'Evolution of fairness in the one-shot anonymous ultimatum game,' Proceedings of the National Academy of Sciences , vol. 110, no. 7, pp. 2581-2586, 2013.
  41. [52]A. Vidler and T. Walsh, 'Playing games with large language models: Randomness and strategy,' arXiv preprint arXiv:2503.02582 , 2025. · arXiv
  42. [53]Y. Deng, Z. Li, S. Mahadevan, and Z. Song, 'Zero-th order algorithm for softmax attention optimization,' in 2024 IEEE International Conference on Big Data (BigData) . IEEE, 2024, pp. 24-33.
  43. [54]Z. Alalawi, T. A. Han, Y. Zeng, and A. Elragig, 'Pathways to good healthcare services and patient satisfaction: An evolutionary game theoretical approach,' in Artificial Life Conference Proceedings . MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info . . . , 2019, pp. 135-142.
  44. [55]H. Chen and J. Hou, 'Intelligent data governance: building an enterprise data management system using kg and llm,' in Proceedings of the 2024 International Conference on Cloud Computing and Big Data , 2024, pp. 266-271.
  45. [56]M. Kinniment, L. J. K. Sato, H. Du, B. Goodrich, M. Hasin et al. , 'Evaluating Language-Model Agents on Realistic Autonomous Tasks,' Jan. 2024. [Online]. Available: http://arxiv.org/abs/2312.11671 · arXiv
  46. [58]J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin, 'Large language models for mathematical reasoning: Progresses and challenges,' arXiv preprint arXiv:2402.00157 , 2024. · arXiv
  47. [59]E. H. Hagen and P. Hammerstein, 'Game theory and human evolution: A critique of some recent interpretations of experimental games,' Theoretical population biology , vol. 69, no. 3, pp. 339-348, 2006.
  48. [60]P. Richerson, R. Baldini, A. V. Bell, K. Demps, K. Frost, V. Hillis, S. Mathew, E. K. Newton, N. Naar, L. Newson et al. , 'Cultural group selection plays an essential role in explaining human cooperation: A sketch of the evidence,' Behavioral and Brain Sciences , vol. 39, p. e30, 2016.
  49. [61]J. C. van den Bergh and J. M. Gowdy, 'A group selection perspective on economic behavior, institutions and organizations,' Journal of Economic Behavior & Organization , vol. 72, no. 1, pp. 1-20, 2009.
  50. [62]R. Ali, F. Caso, C. Irwin, and P. Li` o, 'Entropy-lens: The information signature of transformer computations,' arXiv preprint arXiv:2502.16570 , 2025. · arXiv
  51. [63]B. El, D. Choudhury, P. Li` o, and C. K. Joshi, 'Towards mechanistic interpretability of graph transformers via attention graphs,' arXiv preprint arXiv:2502.12352 , 2025. · arXiv

Cited by

Sorted by citations

Authors
Buscemi, A., Proverbio, D., Bova, P., Balabanova, N., Bashir, A., Cimpeanu, T., da Fonseca, H. C., Duong, M. H., Domingos, E. F., Fernandes, A. M., Krellner, M., Ogbo, N. B., Powers, S. T., Santos, F. P., Shamszaman, Z. U., Song, Z., Di Stefano, A., & Han, T. A.
Affiliations
  • Luxembourg Institute of Science and Technology (LU)
  • University of Trento (IT)
  • Teesside University (GB)
  • University of Birmingham (GB)
  • University of St Andrews (GB)
  • Instituto de Engenharia de Sistemas e Computadores Investigação e Desenvolvimento (PT)
  • University of Stirling (GB)
  • University of Amsterdam (NL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.08640
arXiv
2504.08640
OpenAlex
W4414829248
Cited by
3
References
0
Citations by year
2026: 1 · 2025: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Teesside University Research Portal (Teesside University) (submittedVersion, cc-by)
  • TeesRep (Teesside University) (submittedVersion, cc-by)
  • VUBIR (Vrije Universiteit Brussel) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Future of Life Institute
  • European Commission (101076926, C645008882-00000055)
  • Engineering and Physical Sciences Research Council (EP/Y008561/1, EP/Y00857X/1, IES-R3-223047)
Topics
  • Social Sciences › Sociology and Political Science › Evolutionary Game Theory and Cooperation
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Access Control and Trust
Keywords
AI regulation, LLM agents, evolutionary game theory, trust in AI, strategic behavior, conditional trust, regulatory incentives, AI safety, repeated games

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review