CoChat
Sign Up
PreprintCited by 8Ethics and Social Impacts of AIMulti-Agent Systems and NegotiationAccess Control and Trust

Infrastructure for AI Agents

Open access
Chan, Alan, Wei, Kevin, Huang, Sihao, Rajkumar, Nitarshan, Perrier, Elija, Lazar, Seth, Hadfield, Gillian K., Anderljung, Markus·2025·arXiv
DOI 10.48550/arxiv.2501.10114arXiv 2501.10114
Open this paper in CoChat — free accountOpen PDF

Abstract

AI agents plan and execute interactions in open-ended environments. For example, OpenAI's Operator can use a web browser to do product comparisons and buy online goods. Much research on making agents useful and safe focuses on directly modifying their behaviour, such as by training them to follow user instructions. Direct behavioural modifications are useful, but do not fully address how heterogeneous agents will interact with each other and other actors. Rather, we will need external protocols and systems to shape such interactions. For instance, agents will need more efficient protocols to communicate with each other and form agreements. Attributing an agent's actions to a particular human or other legal entity can help to establish trust, and also disincentivize misuse. Given this motivation, we propose the concept of \textbf{agent infrastructure}: technical systems and shared protocols external to agents that are designed to mediate and influence their interactions with and impacts on their environments. Just as the Internet relies on protocols like HTTPS, our work argues that agent infrastructure will be similarly indispensable to ecosystems of agents. We identify three functions for agent infrastructure: 1) attributing actions, properties, and other information to specific agents, their users, or other actors; 2) shaping agents' interactions; and 3) detecting and remedying harmful actions from agents. We provide an incomplete catalog of research directions for such functions. For each direction, we include analysis of use cases, infrastructure adoption, relationships to existing (internet) infrastructure, limitations, and open questions. Making progress on agent infrastructure can prepare society for the adoption of more advanced agents.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.10114

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • New infrastructure might be needed to reap the benefits and manage the risks of multi-agent systems.

    “Just as new infrastructure was needed to enable the internet (e.g., TCP/IP, HTTP) and secure it (e.g., SSL), so too might new infrastructure be needed to reap the benefits and manage the risks of multi-agent systems (Chan et al., 2025).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Chan et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 42

  1. Cem Anil, Esin Durmus, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, James Sully, Alex Tamkin, Tamera Lanham, Karina Nguyen, Tomasz Korbak, Jared Kaplan, Deep Ganguli, Samuel R. Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud. Many-shot jailbreaking. In The thirty-eighth annual conference on neural information processing systems, 2024.
  2. Mark Bagnoli and Barton L. Lipman. Provision of Public Goods: Fully Implementing the Core through Private Contributions. The Review of Economic Studies, 56(4), October 1989. ISSN 0034-6527.
  3. Jake A. Berkowsky and Thaier Hayajneh. Security issues with certificate authorities. In 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), October 2017.
  4. Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Kather- ine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Tegan Maharaj. Harms from Increasingly Agentic Algorithmic Systems. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT '23, New York, NY, USA, June 2023. Association for Computing Machinery.
  5. Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, and Markus Anderljung. Visibility into AI Agents. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, New York, NY, USA, June 2024a. Association for Computing Machinery.
  6. Bryan H. Choi. The Anonymous Internet. Maryland Law Review, 72, 2012.
  7. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Rein- forcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30.
  8. Sasha Costanza-Chock, Inioluwa Deborah Raji, and Joy Buolamwini. Who Audits the Auditors? Recom- mendations from a field scan of the algorithmic auditing ecosystem. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, New York, NY, USA, June 2022. Association for Computing Machinery.
  9. Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel. Cooperative AI: machines must learn to find common ground. Nature, 593(7857), 2021.
  10. Benjamin Edelman. Adverse selection in online "trust" certifications. In Proceedings of the 11th Interna- tional Conference on Electronic Commerce, ICEC '09, New York, NY, USA, August 2009. Association for Computing Machinery.
  11. James D. Fearon. Rationalist Explanations for War. International Organization, 49(3), 1995. ISSN 0020- 8183.
  12. Kate Goddard, Abdul Roudsari, and Jeremy C Wyatt. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), January 2012. ISSN 1067-5027.
  13. Ben Green. The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, July 2022. ISSN 0267-3649.
  14. Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast. arXiv preprint arXiv:2402.08567, 2024. · arXiv
  15. Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative Inverse Reinforce- ment Learning. In Advances in Neural Information Processing Systems, volume 29.
  16. Md Imran Hossen and Xiali Hei. A Low-Cost Attack against the hCaptcha System. In 2021 IEEE Security and Privacy Workshops (SPW), May 2021.
  17. Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-Play " for zero-shot coordina- tion. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of ICML'20. JMLR.org, July 2020.
  18. Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I. Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. Collective Constitutional AI: Aligning a Language Model with Public Input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, New York, NY, USA, June 2024. Association for Computing Machinery.
  19. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, April 2024.
  20. Deborah G. Johnson. Software Agents, Anticipatory Ethics, and Accountability. In Gary E. Marchant, Braden R. Allenby, and Joseph R. Herkert (eds.), The Growing Gap Between Emerging Technologies and Legal-Ethical Oversight: The Pacing Problem. Springer Netherlands, Dordrecht, 2011.
  21. Ruogu Kang, Stephanie Brown, and Sara Kiesler. Why do people seek anonymity on the internet? informing policy and design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '13, New York, NY, USA, April 2013. Association for Computing Machinery.
  22. Rob Kling, Ya-ching Lee, Al Teich, and Mark S Frankel. Assessing anonymous communication on the internet: Policy deliberations. The Information Society, 15(2), 1999.
  23. Adam Lerer and Alexander Peysakhovich. Learning Existing Social Conventions via Observationally Aug- mented Self-Play. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES '19, New York, NY, USA, January 2019. Association for Computing Machinery.
  24. Henry Lieberman. Autonomous interface agents. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, CHI '97, New York, NY, USA, March 1997. Association for Computing Machinery.
  25. Pattie Maes. Agents that reduce work and information overload. Communications of the ACM, 37(7), July 1994. ISSN 0001-0782.
  26. Pattie Maes. Artificial life meets entertainment: lifelike autonomous agents. Communications of the ACM, 38(11), November 1995. ISSN 0001-0782.
  27. Ashwin Jacob Mathew. Where in the world is the internet? Locating political power in internet infrastructure. PhD thesis, University of California, Berkeley, 2014.
  28. William L Oberkampf and Christopher J Roy. Verification and validation in scientific computing. Cambridge university press, 2010.
  29. Elija Perrier. Classical and Language Model Multi-Agent Infrastructure: A Comparative Review. Forth- coming, 2025.
  30. Stefan Poslad. Specifying protocols for multi-agent systems interaction. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 2(4), 2007.
  31. Robert Powell. War as a Commitment Problem. International Organization, 60(1), 2006. ISSN 0020-8183.
  32. Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchin- son, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, New York, NY, USA, January 2020. Association for Computing Machinery.
  33. Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, AIES '22, New York, NY, USA, July 2022. Association for Computing Machinery.
  34. Stuart J. Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. 4 edition, 2021.
  35. LINDA J. Skitka, KATHLEEN L. Mosier, and MARK Burdick. Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), November 1999. ISSN 1071-5819.
  36. Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018.
  37. Rory Van Loo. Digital Market Perfection. Michigan Law Review, 117(5), January 2019. ISSN 0026-2234.
  38. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024. · arXiv
  39. Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Soft Self-Consistency Improves Language Models Agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Bangkok, Thailand, August 2024a. Association for Computational Linguistics.
  40. Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts, November 2024.
  41. Michael Wooldridge. An Introduction to Multiagent Systems. John Wiley & Sons, 2009.
  42. Jonathan L. Zittrain. The Generative Internet. Harvard Law Review, 119(7), 2006. ISSN 0017-811X.

Cited by

Sorted by citations

Related

Authors
Chan, A., Wei, K., Huang, S., Rajkumar, N., Perrier, E., Lazar, S., Hadfield, G. K., & Anderljung, M.
Affiliations
  • Harvard University (US)
  • University of Oxford (GB)
  • University of Cambridge (GB)
  • Australian National University (AU)
  • Johns Hopkins University (US)
  • Centre for the Governance of AI (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2501.10114
arXiv
2501.10114
OpenAlex
W4406603985
Cited by
4
References
0
Citations by year
2026: 2 · 2025: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University) (public-domain)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Social Sciences › Sociology and Political Science › Access Control and Trust
Keywords
AI agents, agent infrastructure, agent interaction protocols, agent safety, trust in AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review