CoChat
Sign Up
Cited by 63Adversarial Robustness in Machine LearningPrivacy-Preserving Technologies in DataPrivacy, Security, and Data Protection

The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies

Open access
He, Feng, Zhu, Tianqing, Ye, Dayong, Liu, Bo, Zhou, Wanlei, Yu, Philip S.·2026·ACM Computing Surveys
DOI 10.1145/3773080

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Inspired by the rapid development of Large Language Models (LLMs), LLM agents have evolved to perform complex tasks. LLM agents are now extensively applied across various domains, handling vast amounts of data to interact with humans and execute tasks. The widespread applications of LLM agents demonstrate their significant commercial value; however, they also expose security and privacy vulnerabilities. At the current stage, comprehensive research on the security and privacy of LLM agents is highly needed. This survey aims to provide a comprehensive overview of the newly emerged privacy and security issues faced by LLM agents. We begin by introducing the fundamental knowledge of LLM agents, followed by a categorization and analysis of the threats. We then discuss the impacts of these threats on humans, environment, and other agents. Subsequently, we review existing defensive strategies, and finally explore future trends. Additionally, the survey incorporates diverse case studies to facilitate a more accessible understanding. By highlighting these critical security and privacy issues, the survey seeks to stimulate future research towards enhancing the security and privacy of LLM agents, thereby increasing their reliability and trustworthiness in future applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3773080

Cites

93 of 107 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Agents introduce new attack surfaces: prompt injection via untrusted observations, tool hijacking, and emergent collusion in multi-agent settings.

    “They introduce new attack surfaces: prompt injection via untrusted observations [88], tool hijacking, and emergent collusion in multiagent settings [22,26].”

    From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [26]

    The Functional Manipulation section (pp. 15–16) describes attackers redirecting agents’ actions through untrusted tools, supporting the tool-hijacking part of the claim. The Impact to Other Agents (p. 18) and Multi-Agent System (p. 29) sections describe malicious agents influencing others and spreading misinformation, but do not establish emergent collusion. The prompt-injection phrase has a separate citation in the citing sentence. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Multi-agent systems amplify cost (more LLM calls per task), latency (sequential coordination), and attack surface (each inter-agent message is a prompt-injection vector).

    “However, multi-agent systems amplify cost (more LLM calls per task), latency (sequential coordination), and attack surface (each inter-agent message is a prompt-injection vector [26,88]).”

    From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [26]

    Section 6.2 (pp. 28–29) says inter-agent communication can amplify security problems, and Section 4.3 (p. 18) describes malicious agents influencing others through crafted information. The source does not establish higher cost from additional LLM calls or latency from sequential coordination, nor does it say that every inter-agent message is a prompt-injection vector. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Across the full 153-paper corpus, only five studies report observing a compound failure mode at all.

    “across the full 153-paper corpus, only five studies (≈3%)- [22,71,125] for error propagation and [26,87] for tool-use amplification-report observing a compound failure mode at all”

    From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [26]

    The Introduction (p. 2) discusses cascading effects from LLM-agent errors, but does not count studies observing compound failures. Searches for “153,” “five studies,” “compound failure,” and the named failure types found no passage establishing a 153-paper corpus or the claimed five-study tally. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. [12][12] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 21) . 2633-2650. https://www.usenix.org/conference/usenixsecurity21/presentation/carliniextracting -
  2. [16][16] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023. MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots . arXiv:2307.08715 http://arxiv.org/abs/2307.08715 - · arXiv
  3. [36][36] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions . arXiv:2311.05232 http://arxiv.org/abs/2311.05232 - · arXiv
  4. [38][38] Shotaro Ishihara. 2023. Training Data Extraction From Pre-trained Language Models: A Survey. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) (Toronto, Canada), Anaelia Ovalle, Kai-Wei Chang, Ninareh Mehrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Verma, Trista Cao, Anoop Kumar, and Rahul Gupta (Eds.). Association for Computational Linguistics, 260-275. https://aclanthology.org/2023.trustnlp-1.23 -
  5. [37][37] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. 2024. Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training . arXiv:2401.05566 http://arxiv.org/abs/2401.05566 - · arXiv
  6. [40][40] Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards Mitigating Hallucination in Large Language Models via Self-Reflection. arXiv preprint arXiv:2310.06271 (2023). https://arxiv.org/abs/2310.06271 - · arXiv
  7. [43][43] Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2023. ProPILE: Probing Privacy Leakage in Large Language Models . arXiv:2307.01881 http://arxiv.org/abs/2307.01881 - · arXiv
  8. [47][47] Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023. Multi-Step Jailbreaking Privacy Attacks on ChatGPT . arXiv:2304.05197 http://arxiv.org/abs/2304.05197 - · arXiv
  9. [51][51] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id=7Jwpw4qKkb -
  10. [56][56] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning . arXiv:2308.08747 http://arxiv.org/abs/2308.08747 - · arXiv
  11. [60][60] Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-Flamingo: A Multimodal Medical Few-shot Learner. In Proceedings of the 3rd Machine Learning for Health Symposium (Proceedings of Machine Learning Research, Vol. 225) . PMLR, 353-367. https://proceedings.mlr.press/v225/moor23a.html -
  12. [64][64] Mustafa Safa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. 2023. Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning . arXiv:2305.11759 https: //arxiv.org/abs/2305.11759 Proc. ACM Meas. Anal. Comput. Syst., Vol. 37, No. 4, Article 111. Publication date: August 2018. - · arXiv
  13. [67][67] Liangzu Peng, Paris Giampouras, and Rene Vidal. 2023. The Ideal Continual Learner: An Agent That Never Forgets. In Proceedings of the 40th International Conference on Machine Learning . PMLR, 27585-27610. https://proceedings.mlr.press/v202/peng23a.html -
  14. [80][80] Thomas Vakili, Anastasios Lamproudis, Aron Henriksson, and Hercules Dalianis. 2022. Downstream Task Performance of BERT Models Pre-Trained Using Automatically De-Identified Clinical Data. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (Marseille, France), Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (Eds.). European Language Resources Association, 4245-4252. https://aclanthology.org/2022.lrec-1.451 -
  15. [79][79] Jean-Baptiste Truong, Pratyush Maini, Robert J. Walls, and Nicolas Papernot. 2021. Data-Free Model Extraction. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 4769-4778. https://ieeexplore.ieee.org/document/9577784 -
  16. [86][86] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2023. A Survey on Large Language Model Based Autonomous Agents . arXiv:2308.11432 http://arxiv.org/abs/2308.11432 - · arXiv
  17. [88][88] Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. 2024. Large Language Models for Education: A Survey and Outlook . arXiv:2403.18105 https://arxiv.org/abs/2403.18105 - · arXiv
  18. [93][93] Zhenhua Wang, Wei Xie, Kai Chen, Baosheng Wang, Zhiwen Gui, and Enze Wang. 2023. Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models . arXiv:2308.11521 http://arxiv.org/abs/2308.11521 - · arXiv
  19. [106][106] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (Red Hook, NY, USA) (NIPS '23) . Curran Associates Inc., 11809-11822. -
  20. [113][113] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. 2021. Differentially Private Fine-tuning of Language Models. In International Conference on Learning Representations . https://openreview.net/forum?id=Q42f0dfjECO -
  21. [114][114] Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts . arXiv:2309.10253 http://arxiv.org/abs/2309.10253 - · arXiv
  22. [119][119] Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. 2024. The Knowledge Alignment Problem: Bridging Human and External Knowledge for Large Language Models . arXiv:2305.13669 https://arxiv.org/abs/2305.13669 - · arXiv
  23. [128][128] Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2025. PoisonedRAG: Knowledge Poisoning Attacks to Retrieval-Augmented Generation of Large Language Models. In 34th USENIX Security Symposium (USENIX Security 25) . 3827-3844.

Cited by

Sorted by citations

Related

Authors
He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., & Yu, P. S.
Affiliations
  • University of Technology Sydney (AU)
  • City University of Macau (MO)
  • University of Illinois Chicago (US)
ORCID
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2025-12-09
Volume
58
Issue
6
Pages
1-36
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3773080
OpenAlex
W4401201613
Cited by
58
References
93
Citations by year
2026: 43 · 2025: 13 · 2024: 2
FWCI
22.46
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • ACM Computing Surveys (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Fundo para o Desenvolvimento das Ciências e da Tecnologia (FDCT), Macau SAR (0080/2024/RIA2)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
Keywords
LLM agents, large language models, security threats, privacy risks, threat categorization, defense strategy, trustworthiness, human-agent interaction, case studies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review