CoChat
Sign Up
Cited by 24Ethics and Social Impacts of AIDigitalization, Law, and RegulationSafety Systems Engineering in Autonomy

Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective

Open access
Gabison, Garry A., Xian, R. Patrick·2025·Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025)
DOI 10.18653/v1/2025.realm-1.9

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Agentic systems powered by large language models (LLMs) are becoming progressively more complex and capable.Their increasing agency and expanding deployment settings attract growing attention to effective governance policies, monitoring, and control protocols.Based on the emerging landscape of the agentic market, we analyze potential liability issues arising from the delegated use of LLM agents and their extended systems through a principalagent perspective.Our analysis complements existing risk-based studies on artificial agency and covers the spectrum of important aspects of the principal-agent relationship and their potential consequences at deployment.Furthermore, we motivate method developments for technical governance along the directions of interpretability and behavior evaluations, reward and conflict management, and the mitigation of misalignment and misconduct through principled engineering of detection and fail-safe mechanisms.By illustrating the outstanding issues in AI liability for LLM-based agentic systems, we aim to inform the system design, auditing, and tracing to enhance transparency and liability attribution.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2025.realm-1.9

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI Agents, particularly those powered by LLMs, inherit intrinsic limitations that impact their reliability, adaptability, and overall trustworthiness in practical deployments.

    “AI Agents, particularly those powered by LLMs, inherit a number of intrinsic limitations that impact their reliability, adaptability, and overall trustworthiness in practical deployments [187-189].”

    From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [189]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 57

  1. Kenneth S Abraham. 2012. The Forms and Functions of Tort Law. Foundation Press.
  2. Kenneth S Abraham and Robert L Rabin. 2019. Auto- mated vehicles and manufacturer responsibility for accidents. Virginia Law Review, 105(1):127-171.
  3. Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Gün- ther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric J. Bigelow, Alexander Pan, Lauro Langosco, and 23 others. 2024. Foundational Chal- lenges in Assuring Alignment and Safety of Large Language Models. Transactions on Machine Learn- ing Research.
  4. Kenneth J Arrow. 1963. Uncertainty and the welfare economics of medical care. The American Economic Review, 53:941-973.
  5. Aaron Baird and Likoebe Maruping. 2021. The Next Generation of Research on IS Use: A Theoretical Framework of Delegation to and from Agentic IS Ar- tifacts. Management Information Systems Quarterly, 45(1):315-341.
  6. Woodrow Barfield. 2018. Liability for autonomous and artificially intelligent robots. Paladyn, Journal of Behavioral Robotics, 9(1):193-203.
  7. Len Bass, Paul Clements, and Rick Kazman. 2021. Soft- ware Architecture in Practice. Addison-Wesley Pro- fessional.
  8. Marcel Binz and Eric Schulz. 2023. Turning large lan- guage models into cognitive models. In The Twelfth International Conference on Learning Representa- tions.
  9. Joshua A. Blonz. 2023. The Costs of Misaligned Incen- tives: Energy Inefficiency and the Principal-Agent Problem. American Economic Journal: Economic Policy, 15(3):286-321. · DOI
  10. Rodolfo A Camacho. 1993. How to avoid negligent hiring litigation. Whittier Law Review, 14:787-808.
  11. Emanuela Carbonara, Alice Guerra, and Francesco Parisi. 2016. Sharing residual liability: the cheapest cost avoider revisited. The Journal of Legal Studies, 45(1):173-201. · DOI
  12. Bryan Casey. 2019. Robot Ipsa Loquitur. Georgetown Law Journal, 108(2):225-286.
  13. Frank J Cavico, Bahaudin G Mujtaba, Marissa Samuel, and Stephen C Muffler. 2016. The tort of negligence in employment hiring, supervision, and retention. American Journal of Business and Society, 1(4):205.
  14. Bryan H. Choi. 2020. Software as a Profession. Har- vard Journal of Law & Technology (Harvard JOLT), 33(2):557-638.
  15. Samir Chopra and Laurence F. White. 2011. A Legal Theory for Autonomous Artificial Agents. University of Michigan Press, Ann Arbor.
  16. Peter Cihon. 2024. Chilling autonomy: Policy enforce- ment for human oversight of AI agents. In 41st In- ternational Conference on Machine Learning, Work- shop on Generative AI and Law.
  17. Ronald Harry Coase. 1960. The problem of social cost. The Journal of Law and Economics, 3:1-44.
  18. A. Feder Cooper, Christopher A. Choquette-Choo, Mi- randa Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, Nicole Mitchell, Percy Liang, Daniel E. Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, and 16 others. 2024. Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice. arXiv preprint. ArXiv:2412.06966 · DOI
  19. Jessica Dai. 2024. Position: Beyond Personhood: Agency, Accountability, and the Limits of Anthropo- morphic Ethical Analysis. In Forty-first International Conference on Machine Learning.
  20. Parashar Das. 2025. Agency in Artificial Intelligence Systems. arXiv preprint. ArXiv:2502.10434 [cs]. · DOI
  21. Mihailis E. Diamantis. 2023. Vicarious Liability for AI. Indiana Law Journal, 99(1):317-334.
  22. Manfred Diaz, Joel Z. Leibo, and Liam Paull. 2024. Milnor-Myerson Games and The Principles of Artifi- cial Principal-Agent Problems. In Finding the Frame: An RLC Workshop for Examining Conceptual Frame- works.
  23. Kathleen M. Eisenhardt. 1989. Agency Theory: An Assessment and Review. The Academy of Manage- ment Review, 14(1):57-74. Publisher: Academy of Management. · DOI · Paper page
  24. Eugene F. Fama and Michael C. Jensen. 1983. Agency Problems and Residual Claims. The Journal of Law & Economics, 26(2):327-349. Publisher: [University of Chicago Press, Booth School of Business, Univer- sity of Chicago, University of Chicago Law School].
  25. Garry A. Gabison and Miriam C. Buiten. 2020. Plat- form Liability in Copyright Enforcement. Columbia Science and Technology Law Review, 21(2):237-281.
  26. Michael C. Gemignani. 1980. Product Liability and Software. Rutgers Computer & Technology Law Journal, 8(2):173-204.
  27. Bengt Holmström. 1979. Moral hazard and observabil- ity. The Bell journal of economics, 10:74-91.
  28. Song Jiang, Da Ju, Andrew Cohen, Sasha Mitts, Aaron Foss, Justine T. Kao, Xian Li, and Yuandong Tian. 2024. Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning. In NeurIPS 2024 Workshop on Adaptive Foundation Models.
  29. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bham- bri, Lucas Paul Saldyt, and Anil B. Murthy. 2024. Position: LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks. In Forty-first Interna- tional Conference on Machine Learning.
  30. Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park. 2024. MDAgents: An Adaptive Collabora- tion of LLMs for Medical Decision-Making. In The Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems.
  31. Jean-Jacques Laffont and David Martimort. 2002. The Theory of Incentives: The Principal-Agent Model. Princeton University Press, Princeton, N.J. Leon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. 2024. When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  32. Saul Levmore. 1990. Probabilistic recoveries, restitu- tion, and recurring wrongs. The Journal of Legal Studies, 19(S2):691-726.
  33. Xinzhe Li. 2025. A Review of Prominent Paradigms for LLM-Based Agents: Tool Use, Planning (Including RAG), and Feedback Learning. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9760-9779, Abu Dhabi, UAE. As- sociation for Computational Linguistics.
  34. Ryan Y. Lin, Siddhartha Ojha, Kevin Cai, and Maxwell Chen. 2024. Strategic Collusion of LLM Agents: Market Division in Multi-Commodity Competitions. In Language Gamification -NeurIPS 2024 Workshop.
  35. Brian Lubars and Chenhao Tan. 2019. Ask not what AI can do, but what AI should do: Towards a frame- work of task delegability. In Advances in Neural Information Processing Systems, volume 32.
  36. James Mattingly and Beba Cibralic. 2025. Machine Agency. MIT Press.
  37. Natalie Maus, Patrick Chao, Eric Wong, and Jacob R. Gardner. 2023. Black Box Adversarial Prompting for Foundation Models. In The Second Workshop on New Frontiers in Adversarial Machine Learning.
  38. Neil J. Mitchell. 2021. Why Delegate? Oxford Univer- sity Press, New York, NY.
  39. Barry M. Mitnick. 1992. The Theory of Agency and Or- ganizational Analysis. In Ethics and Agency Theory: An Introduction. Oxford University Press, Oxford, United Kingdom.
  40. Roderick Munday. 2022. Agency: Law and Principles, 4th edition. Oxford University Press, Oxford, United Kingdom ; New York, NY.
  41. Melissa Z. Pan, Mert Cemri, Lakshya A. Agrawal, Shuyi Yang, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Kannan Ramchandran, Dan Klein, Joseph E. Gonzalez, Matei Zaharia, and Ion Stoica. 2025. Why Do Multiagent Systems Fail? In ICLR 2025 Workshop on Building Trust in Language Models and Applications.
  42. Karl Popp. 2011. Software Industry Business Mod- els. IEEE Software, 28(4):26-30. Conference Name: IEEE Software. · DOI
  43. Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2025. Scaling Large Language Model-based Multi- Agent Collaboration. In The Thirteenth International Conference on Learning Representations.
  44. Elspeth Reid. 1999. Liability for Dangerous Activities: A Comparative Analysis. International & Compara- tive Law Quarterly, 48(4):731-756. · DOI
  45. Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn. 2024. Large Language Models can Strategically De- ceive their Users when Put Under Pressure. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
  46. Lisa Soder, Julia Smakman, Connor Dunlop, Weiwei Pan, Siddharth Swaroop, and Noam Kolt. 2024. Lev- els of Autonomy: Liability in the age of AI Agents. In Workshop on Socially Responsible Language Mod- elling Research.
  47. Alan Sykes. 1984. The Economics of Vicarious Li- ability. Yale Law Journal. Accepted: 2021-11- 26T12:26:37Z.
  48. Rakshit Trivedi, Nikhil Chandak, Andrei Ioan Mure- sanu, Shuhui Zhu, Atrisha Sarkar, Joel Z. Leibo, Dy- lan Hadfield-Menell, and Gillian K. Hadfield. 2024. Altared Environments: The Role of Normative In- frastructure in AI Alignment. In Agentic Markets Workshop at ICML 2024.
  49. Yi-Lin Tuan, Connor Pryor, Wenhu Chen, Lise Getoor, and William Yang Wang. 2021. Local Explanation of Dialogue Response Generation. In Advances in Neural Information Processing Systems, volume 34, pages 404-416.
  50. Jacob Turner. 2018. Robot Rules: Regulating Artificial Intelligence. Palgrave Macmillan, New York, NY.
  51. Yoav Vardi and Ely Weitz. 2016. Misbehavior in Orga- nizations: A Dynamic Approach, 2nd edition. Rout- ledge.
  52. Caroline Wang, Arrasy Rahman, Ishan Durugkar, Elad Liebman, and Peter Stone. 2024a. N-agent Ad Hoc Teamwork. Advances in Neural Information Process- ing Systems, 37:111832-111862.
  53. Qian Wang, Tianyu Wang, Zhenheng Tang, Qinbin Li, Nuo Chen, Jingsheng Liang, and Bingsheng He. 2025. All It Takes Is One Prompt: An Autonomous LLM-MA System. In ICLR 2025 Workshop on Foun- dation Models in the Wild.
  54. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024b. Exe- cutable code actions elicit better LLM agents. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of ICML'24, pages 50208-50232, Vienna, Austria. JMLR.org.
  55. Marcus Williams, Micah Carroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy, and Anca Dra- gan. 2025. On Targeted Manipulation and Decep- tion when Optimizing LLMs for User Feedback. In The Thirteenth International Conference on Learning Representations.
  56. Yijia Xiao, Edward Sun, Di Luo, and Wei Wang. 2025. TradingAgents: Multi-Agents LLM Financial Trad- ing Framework. In The First MARW: Multi-Agent AI in the Real World Workshop at AAAI 2025.
  57. Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Shiyang Lai, Kai Shu, Jindong Gu, Adel Bibi, Ziniu Hu, David Jurgens, James Evans, Philip Torr, Bernard Ghanem, and Guohao Li. 2024. Can Large Language Model Agents Simulate Human Trust Behavior? In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems.

Cited by

Sorted by citations

Related

Authors
Gabison, G. A., & Xian, R. P.
Affiliations
  • Queen Mary University of London (GB)
  • University of California, San Francisco (US)
ORCID
Venue
Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025)
Publisher
Association for Computational Linguistics
Published
2025
Pages
109-130
DOI
10.18653/v1/2025.realm-1.9
OpenAlex
W4412944208
Cited by
19
References
0
Citations by year
2026: 11 · 2025: 8
FWCI
19.55
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Workshop for Research on Agent Language Models (REALM) (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Keywords
AI liability, large language models, principal-agent relationship, AI governance, liability allocation, interpretability, behavioral assessment, fail-safe mechanism

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review