CoChat
Sign Up
Cited by 34Access Control and TrustAdversarial Robustness in Machine LearningMulti-Agent Systems and Negotiation

A Survey on Trustworthy LLM Agents: Threats and Countermeasures

Open access
Yu, Miao, Meng, Fanci, Zhou, Xinyun, Wang, Shilong, Mao, Junyuan, Pan, Linsey, Chen, Tianlong, Wang, Kun +4 more·2025·Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
DOI 10.1145/3711896.3736561
Open this paper in CoChat — free accountOpen PDF

Abstract

With the rapid evolution of Large Language Models (LLMs), LLMbased agents and Multi-agent Systems (MAS) have significantly expanded the capabilities of LLM ecosystems.This evolution stems from empowering LLMs with additional modules such as memory, tools, environment, and even other agents.However, this advancement has also introduced more complex issues of trustworthiness, which previous research focusing solely on LLMs could not cover.In this survey, we propose the TrustAgent framework, a comprehensive study on the trustworthiness of agents, characterized by modular taxonomy, multi-dimensional connotations, and * Miao Yu and Fanci Meng contribute equally to this paper.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3711896.3736561

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI Agents are not yet safe or verifiable enough for deployment in handling or managing critical infrastructure.

    “AI Agents are not yet safe or verifiable enough for deployment in handling or managing critical infrastructure [203].”

    From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [203]

    Section 3.2 describes trustworthiness risks in physical environments and says existing solutions are fragmented. Section 3.2.1 discusses safety and code-verification work for industrial control, but does not conclude that agents are currently too unsafe or unverifiable to deploy in critical infrastructure. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 38 · Cited by 34

Research works printed but not linked: 104

  1. Divyansh Agarwal, Alexander R Fabbri, Ben Risher, Philippe Laban, Shafiq Joty, and Chien-Sheng Wu. 2024. Prompt Leakage effect and defense strategies for multi-turn LLM interactions. arXiv preprint arXiv:2404.16251 (2024). · arXiv
  2. Maya Anderson, Guy Amit, and Abigail Goldsteen. 2024. Is my data in your retrieval database? membership inference attacks against retrieval augmented generation. arXiv preprint arXiv:2405.20446 (2024). · arXiv
  3. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Due- nas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. 2024. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024 (2024). · arXiv
  4. Md Ahsan Ayub and Subhabrata Majumdar. 2024. Embedding-based classifiers can detect prompt injection attacks. arXiv preprint arXiv:2410.22284 (2024). · arXiv
  5. Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. 2023. Abusing images and sounds for indirect instruction injection in multi-modal LLMs. arXiv preprint arXiv:2307.10490 (2023). · arXiv
  6. Bocheng Chen, Guangjing Wang, Hanqing Guo, Yuanda Wang, and Qiben Yan. 2023. Understanding multi-turn toxic behaviors in open-domain chatbots. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses. 282-296.
  7. Jizhou Chen and Samuel Lee Cong. 2025. AgentGuard: Repurposing Agen- tic Orchestrator for Safety Evaluation of Tool Orchestration. arXiv preprint arXiv:2502.09809 (2025). · arXiv
  8. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2024. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363 (2024). · arXiv
  9. Zhaorun Chen, Zhuokai Zhao, Wenjie Qu, Zichen Wen, Zhiguang Han, Zhihong Zhu, Jiaheng Zhang, and Huaxiu Yao. 2024. Pandora: Detailed llm jailbreaking via collaborated phishing agents with decomposed reasoning. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models.
  10. Wen Cheng, Ke Sun, Xinyu Zhang, and Wei Wang. 2024. Security Attacks on LLM-based Code Completion Tools. arXiv preprint arXiv:2408.11006 (2024). · arXiv
  11. Yixin Cheng, Markos Georgopoulos, Volkan Cevher, and Grigorios G Chrysos. 2024. Leveraging the context through multi-round interactions for jailbreaking attacks. arXiv preprint arXiv:2402.09177 (2024). · arXiv
  12. Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, and Jong C Park. 2024. Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations. arXiv preprint arXiv:2404.13948 (2024). · arXiv
  13. Gobinda Chowdhury and Sudatta Chowdhury. 2024. AI-and LLM-driven search tools: A paradigm shift in information access for education and research. Journal of Information Science (2024), 01655515241284046.
  14. Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents. arXiv preprint arXiv:2406.13352 (2024). · arXiv
  15. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2025. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. Advances in Neural Information Processing Systems 37 (2025), 82895-82920.
  16. Diego Dorn, Alexandre Variengien, Charbel-RaphaÃG , l Segerie, and Vincent Corruble. 2024. Bells: A framework towards future proof benchmarks for the evaluation of llm safeguards. arXiv preprint arXiv:2406.01364 (2024). · arXiv
  17. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mor- datch. 2023. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning.
  18. Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024. Llm agents can autonomously hack websites. arXiv preprint arXiv:2402.06664 (2024). · arXiv
  19. Ivar Frisch and Mario Giulianelli. 2024. LLM agents in interaction: Measuring personality consistency and linguistic alignment in interacting populations of large language models. arXiv preprint arXiv:2402.02896 (2024). · arXiv
  20. Xiaohan Fu, Shuheng Li, Zihan Wang, Yihao Liu, Rajesh K Gupta, Taylor Berg- Kirkpatrick, and Earlence Fernandes. 2024. Imprompter: Tricking LLM Agents into Improper Tool Use. arXiv preprint arXiv:2410.14923 (2024). · arXiv
  21. Xiaohan Fu, Zihan Wang, Shuheng Li, Rajesh K Gupta, Niloofar Mireshghal- lah, Taylor Berg-Kirkpatrick, and Earlence Fernandes. 2023. Misusing tools in large language models with visual adversarial examples. arXiv preprint arXiv:2310.03185 (2023). · arXiv
  22. Yuyou Gan, Yong Yang, Zhe Ma, Ping He, Rui Zeng, Yiming Wang, Qingming Li, Chunyi Zhou, Songze Li, Ting Wang, et al. 2024. Navigating the risks: A survey of security, privacy, and ethics threats in llm-based agents. arXiv preprint arXiv:2411.09523 (2024). · arXiv
  23. Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. 2024. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast. arXiv preprint arXiv:2402.08567 (2024). · arXiv
  24. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024). · arXiv
  25. Lewis Hammond, Alan Chan, Jesse Clifton, Jason Hoelscher-Obermaier, Akbir Khan, Euan McLean, Chandler Smith, Wolfram Barfuss, Jakob Foerster, Tomáš Gavenčiak, et al. 2025. Multi-Agent Risks from Advanced AI. arXiv preprint arXiv:2502.14143 (2025). · arXiv
  26. Feng He, Tianqing Zhu, Dayong Ye, Bo Liu, Wanlei Zhou, and Philip S Yu. 2024. The emerged security and privacy of llm agent: A survey with case studies. arXiv preprint arXiv:2407.19354 (2024). · arXiv
  27. Pengfei He, Yupin Lin, Shen Dong, Han Xu, Yue Xing, and Hui Liu. 2025. Red- Teaming LLM Multi-Agent Systems via Communication Attacks. arXiv preprint arXiv:2502.14847 (2025). · arXiv
  28. Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. 2024. Trustagent: Towards safe and trustworthy llm-based agents. In Findings of the Association for Computational Linguistics: EMNLP 2024. 10000-10016.
  29. Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561 (2024). · arXiv
  30. Korir Nancy Jeptoo and Chengjie Sun. 2024. Enhancing Fake News Detection with Large Language Models Through Multi-agent Debates. In CCF International Conference on Natural Language Processing and Chinese Computing. Springer, 474-486.
  31. Changyue Jiang, Xudong Pan, Geng Hong, Chenfu Bao, and Min Yang. 2024. Rag- thief: Scalable extraction of private data from retrieval-augmented generation applications with agent-based attacks. arXiv preprint arXiv:2411.14110 (2024). · arXiv
  32. Ziyou Jiang, Mingyang Li, Guowei Yang, Junjie Wang, Yuekai Huang, Zhiyuan Chang, and Qing Wang. 2025. Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System. arXiv preprint arXiv:2502.11358 (2025). · arXiv
  33. Tianjie Ju, Yiting Wang, Xinbei Ma, Pengzhou Cheng, Haodong Zhao, Yulong Wang, Lifeng Liu, Jian Xie, Zhuosheng Zhang, and Gongshen Liu. 2024. Flooding spread of manipulated knowledge in llm-based multi-agent communities. arXiv preprint arXiv:2407.07791 (2024). · arXiv
  34. Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. 2023. Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705 (2023). · arXiv
  35. Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx, Shuyan Zhou, Matt Fredrikson, et al. 2024. Refusal-trained llms are easily jailbroken as browser agents. arXiv preprint arXiv:2410.13886 (2024). · arXiv
  36. Ted Kwartler, Matthew Berman, and Alan Aqrawi. 2024. Good Parenting is all you need-Multi-agentic LLM Hallucination Mitigation. arXiv preprint arXiv:2410.14262 (2024). · arXiv
  37. Ohjoon Kwon, Donghyeon Jeon, Nayoung Choi, Gyu-Hwung Cho, Changbong Kim, Hyunwoo Lee, Inho Kang, Sun Kim, and Taiwoo Park. 2024. SLM as Guardian: Pioneering AI Safety with Small Language Models. arXiv preprint arXiv:2405.19795 (2024). · arXiv
  38. Lucio La Cava and Andrea Tagarelli. 2024. Safeguarding Decentralized Social Media: LLM Agents for Automating Community Rule Compliance. arXiv preprint arXiv:2409.08963 (2024). · arXiv
  39. Donghyun Lee and Mo Tiwari. 2024. Prompt infection: Llm-to-llm prompt injection within multi-agent systems. arXiv preprint arXiv:2410.07283 (2024). · arXiv
  40. Ang Li, Yin Zhou, Vethavikashini Chithrra Raghuram, Tom Goldstein, and Micah Goldblum. 2025. Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks. arXiv preprint arXiv:2502.08586 (2025). · arXiv
  41. Haoran Li, Mingshi Xu, and Yangqiu Song. 2023. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. arXiv preprint arXiv:2305.03010 (2023). · arXiv
  42. Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459 (2024). · arXiv
  43. Guang Lin and Qibin Zhao. 2024. Large Language Model Sentinel: LLM Agent for Adversarial Purification. arXiv preprint arXiv:2405.20770 (2024). · arXiv
  44. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957 (2024). · arXiv
  45. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthy LLMs: A survey and guideline for evaluating large language mod- els' alignment. arXiv preprint arXiv:2308.05374 (2023). · arXiv
  46. Zihan Liu, Ruinan Zeng, Dongxia Wang, Gengyun Peng, Jingyi Wang, Qiang Liu, Peiyu Liu, and Wenhai Wang. 2024. Agents4PLC: Automating Closed-loop PLC Code Generation and Verification in Industrial Control Systems using LLM-based Agents. arXiv preprint arXiv:2410.14209 (2024). · arXiv
  47. Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. 2024. The land- scape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. arXiv preprint arXiv:2404.11584 (2024). · arXiv
  48. John X Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. 2023. Text embeddings reveal (almost) as much as text. arXiv preprint arXiv:2310.06816 (2023). · arXiv
  49. Subhabrata Mukherjee, Paul Gamble, Markel Sanz Ausin, Neel Kant, Kriti Aggar- wal, Neha Manjunath, Debajyoti Datta, Zhengliang Liu, Jiayuan Ding, Sophia Busacca, et al. 2024. Polaris: A safety-focused llm constellation architecture for healthcare. arXiv preprint arXiv:2403.13313 (2024). · arXiv
  50. Yuzhou Nie, Zhun Wang, Ye Yu, Xian Wu, Xuandong Zhao, Wenbo Guo, and Dawn Song. 2024. PrivAgent: Agentic-based Red-teaming for LLM Privacy Leakage. arXiv preprint arXiv:2412.05734 (2024). · arXiv
  51. Liviu Panait and Sean Luke. 2005. Cooperative multi-agent learning: The state of the art. Autonomous agents and multi-agent systems 11 (2005), 387-434.
  52. Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. 2024. Self-alignment of large language models via multi- agent social simulation. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
  53. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2023. Identifying the risks of lm agents with an lm-emulated sandbox. arXiv preprint arXiv:2309.15817 (2023). · arXiv
  54. Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2024. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack. arXiv preprint arXiv:2404.01833 (2024). · arXiv
  55. Zhuocheng Shen. 2024. Llm with tools: A survey. arXiv preprint arXiv:2409.18807 (2024). · arXiv
  56. Chengyu Song, Linru Ma, Jianming Zheng, Jinzhi Liao, Hongyu Kuang, and Lin Yang. 2024. Audit-LLM: Multi-Agent Collaboration for Log-based Insider Threat Detection. arXiv preprint arXiv:2408.08902 (2024). · arXiv
  57. Ruoyu Song, Muslum Ozgur Ozmen, Hyungsub Kim, Antonio Bianchi, and Z Berkay Celik. 2024. Enhancing llm-based autonomous driving agents to mitigate perception attacks. arXiv preprint arXiv:2409.14488 (2024). · arXiv
  58. Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Yu Kong, Tianlong Chen, and Huan Liu. 2024. The wolf within: Covert injection of malice into mllm societies via an mllm operative. arXiv preprint arXiv:2402.14859 (2024). · arXiv
  59. Xiangru Tang, Qiao Jin, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, Meng Qu, Yilun Zhao, Jian Tang, Zhuosheng Zhang, et al. 2024. Prioritiz- ing safeguarding over autonomy: Risks of llm agents for science. arXiv preprint arXiv:2402.04247 (2024). · arXiv
  60. Elizaveta Tennant, Stephen Hailes, and Mirco Musolesi. 2024. Moral Alignment for LLM Agents. arXiv preprint arXiv:2410.01639 (2024). · arXiv
  61. Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. 2023. Evil geniuses: Delving into the safety of llm-based agents. arXiv preprint arXiv:2311.11855 (2023). · arXiv
  62. Terry Tong, Jiashu Xu, Qin Liu, and Muhao Chen. 2024. Securing Multi-turn Conversational Language Models From Distributed Backdoor Triggers. arXiv preprint arXiv:2407.04151 (2024). · arXiv
  63. Javal Vyas and Mehmet Mercangöz. 2024. Autonomous Industrial Control using an Agentic Framework with Large Language Models. arXiv preprint arXiv:2411.05904 (2024). · arXiv
  64. Fengxiang Wang, Ranjie Duan, Peng Xiao, Xiaojun Jia, YueFeng Chen, Chong- wen Wang, Jialing Tao, Hang Su, Jun Zhu, and Hui Xue. 2024. Mrj-agent: An ef- fective jailbreak agent for multi-round dialogue. arXiv preprint arXiv:2411.03814 (2024). · arXiv
  65. Haowei Wang, Rupeng Zhang, Junjie Wang, Mingyang Li, Yuekai Huang, Dan- dan Wang, and Qing Wang. 2024. From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection. arXiv preprint arXiv:2412.10198 (2024). · arXiv
  66. Shilong Wang, Guibin Zhang, Miao Yu, Guancheng Wan, Fanci Meng, Chongye Guo, Kun Wang, and Yang Wang. 2025. G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systems. arXiv preprint arXiv:2502.11127 (2025). · arXiv
  67. Yuntao Wang, Yanghe Pan, Quan Zhao, Yi Deng, Zhou Su, Linkang Du, and Tom H Luan. 2024. Large Model Agents: State-of-the-Art, Coopera- tion Paradigms, Security and Privacy, and Future Trends. arXiv preprint arXiv:2409.14457 (2024). · arXiv
  68. Yifei Wang, Dizhan Xue, Shengjie Zhang, and Shengsheng Qian. 2024. Bada- gent: Inserting and activating backdoor attacks in llm agents. arXiv preprint arXiv:2406.03007 (2024). · arXiv
  69. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: En- abling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155 (2023). · arXiv
  70. Yi Wu, Zikang Xiong, Yiran Hu, Shreyash S Iyengar, Nan Jiang, Aniket Bera, Lin Tan, and Suresh Jagannathan. 2024. SELP: Generating safe and effi- cient task plans for robot agents with large language models. arXiv preprint arXiv:2409.19471 (2024). · arXiv
  71. Xun Xian, Ganghua Wang, Xuan Bi, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, and Jie Ding. 2024. On the Vulnerability of Apply- ing Retrieval-Augmented Generation within Knowledge-Intensive Application Domains. arXiv preprint arXiv:2409.17275 (2024). · arXiv
  72. Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. 2024. Certifiably robust rag against retrieval corruption. arXiv preprint arXiv:2405.15556 (2024). · arXiv
  73. Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, et al. 2024. Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning. arXiv preprint arXiv:2406.09187 (2024). · arXiv
  74. Huiyu Xu, Wenhui Zhang, Zhibo Wang, Feng Xiao, Rui Zheng, Yunhe Feng, Zhongjie Ba, and Kui Ren. 2024. Redagent: Red teaming large language models with context-aware autonomous language agent. arXiv preprint arXiv:2407.16667 (2024). · arXiv
  75. Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. 2024. Watch out for your agents! investigating backdoor threats to llm-based agents. arXiv preprint arXiv:2402.11208 (2024). · arXiv
  76. Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. 2024. Embodied multi-modal agent trained by an llm from a parallel textworld. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26275-26285.
  77. Ziyi Yang, Shreyas S Raman, Ankit Shah, and Stefanie Tellex. 2024. Plug in the safety chip: Enforcing constraints for llm-driven robot agents. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 14435-14442.
  78. Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Toolsword: Unveiling safety issues of large language models in tool learning across three stages. arXiv preprint arXiv:2402.10753 (2024). · arXiv
  79. Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, Jing Shao, and Siheng Chen. 2024. SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents. arXiv preprint arXiv:2412.13178 (2024). · arXiv
  80. Miao Yu, Shilong Wang, Guibin Zhang, Junyuan Mao, Chenlong Yin, Qijiong Liu, Qingsong Wen, Kun Wang, and Yang Wang. 2024. Netsafe: Exploring the topological safety of multi-agent networks. arXiv preprint arXiv:2410.15686 (2024). · arXiv
  81. Yinbo Yu, Saihao Yan, Xueyu Yin, Jing Fang, and Jiajia Liu. 2025. BLAST: A Stealthy Backdoor Leverage Attack against Cooperative Multi-Agent Deep Reinforcement Learning based Systems. arXiv preprint arXiv:2501.01593 (2025). · arXiv
  82. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. 2024. R- judge: Benchmarking safety risk awareness for llm agents. arXiv preprint arXiv:2401.10019 (2024). · arXiv
  83. Xiaohan Yuan, Jinfeng Li, Dongxia Wang, Yuefeng Chen, Xiaofeng Mao, Long- tao Huang, Hui Xue, Wenhai Wang, Kui Ren, and Jingyi Wang. 2024. S-eval: Automatic and adaptive test generation for benchmarking safety evaluation of large language models. arXiv preprint arXiv:2405.14191 (2024). · arXiv
  84. Shenglai Zeng, Jiankun Zhang, Pengfei He, Yue Xing, Yiding Liu, Han Xu, Jie Ren, Shuaiqiang Wang, Dawei Yin, Yi Chang, et al. 2024. The good and the bad: Exploring privacy issues in retrieval-augmented generation (rag). arXiv preprint arXiv:2402.16893 (2024). · arXiv
  85. Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. 2024. Autodefense: Multi-agent llm defense against jailbreak attacks. arXiv preprint arXiv:2403.04783 (2024). · arXiv
  86. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691 (2024). · arXiv
  87. Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, and Yang Zhang. 2024. Breaking agents: Compromising autonomous llm agents through malfunction amplification. arXiv preprint arXiv:2407.20859 (2024). · arXiv
  88. Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. 2024. Ufo: A ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939 (2024). · arXiv
  89. Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. arXiv preprint arXiv:2410.02644 (2024). · arXiv
  90. Jiawei Zhang, Chejian Xu, and Bo Li. 2024. Chatscene: Knowledge-enabled safety-critical scenario generation for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15459-15469.
  91. Shiyao Zhang, Yuji Dong, Yichuan Zhang, Terry R Payne, and Jie Zhang. 2024. Large Language Model Assissted Multi-Agent Dialogue for Ontology Alignment. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. 2594-2596.
  92. Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024. Agent-SafetyBench: Evaluating the Safety of LLM Agents. arXiv preprint arXiv:2412.14470 (2024). · arXiv
  93. Zhexin Zhang, Yida Lu, Jingyuan Ma, Di Zhang, Rui Li, Pei Ke, Hao Sun, Lei Sha, Zhifang Sui, Hongning Wang, et al. 2024. Shieldlm: Empowering llms as aligned, customizable and explainable safety detectors. arXiv preprint arXiv:2402.16444 (2024). · arXiv
  94. Zaibin Zhang, Yongting Zhang, Lijun Li, Hongzhi Gao, Lijun Wang, Huchuan Lu, Feng Zhao, Yu Qiao, and Jing Shao. 2024. Psysafe: A comprehensive framework for psychological-based attack, defense, and evaluation of multi-agent system safety. arXiv preprint arXiv:2401.11880 (2024). · arXiv
  95. Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. 2023. Poi- soning retrieval corpora by injecting adversarial passages. arXiv preprint arXiv:2310.19156 (2023). · arXiv
  96. Huichi Zhou, Kin-Hei Lee, Zhonghao Zhan, Yue Chen, and Zhenhao Li. 2025. TrustRAG: Enhancing Robustness and Trustworthiness in RAG. arXiv preprint arXiv:2501.00879 (2025). · arXiv
  97. Xuhui Zhou, Hyunwoo Kim, Faeze Brahman, Liwei Jiang, Hao Zhu, Ximing Lu, Frank Xu, Bill Yuchen Lin, Yejin Choi, Niloofar Mireshghallah, et al. 2024. Haicosystem: An ecosystem for sandboxing safety risks in human-ai interactions. arXiv preprint arXiv:2409.16427 (2024). · arXiv
  98. Yihe Zhou, Tao Ni, Wei-Bin Lee, and Qingchuan Zhao. 2025. A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluations. arXiv preprint arXiv:2502.05224 (2025). · arXiv
  99. Zhenhong Zhou, Zherui Li, Jie Zhang, Yuanhe Zhang, Kun Wang, Yang Liu, and Qing Guo. 2025. CORBA: Contagious Recursive Blocking Attacks on Multi- Agent Systems Based on Large Language Models. arXiv preprint arXiv:2502.14529 (2025). · arXiv
  100. Pengyu Zhu, Zhenhong Zhou, Yuanhe Zhang, Shilinlu Yan, Kun Wang, and Sen Su. 2025. DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agent. arXiv preprint arXiv:2502.12575 (2025). · arXiv
  101. Zihao Zhu, Bingzhe Wu, Zhengyou Zhang, and Baoyuan Wu. 2024. Riskawarebench: Towards evaluating physical risk awareness for high-level planning of llm-based embodied agents. arXiv e-prints (2024), arXiv-2408.
  102. Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning.
  103. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023). · arXiv
  104. Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2024. Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large lan- guage models. arXiv preprint arXiv:2402.07867 (2024). · arXiv

Authors
Yu, M., Meng, F., Zhou, X., Wang, S., Mao, J., Pan, L., Chen, T., Wang, K., Li, X., Zhang, Y., An, B., & Wen, Q.
Affiliations
  • University of Science and Technology of China (CN)
  • Zhejiang University (CN)
  • Salesforce, San Francisco, Illinois, USA
  • Massachusetts Institute of Technology (US)
  • Nanyang Technological University (SG)
  • Rutgers, The State University of New Jersey (US)
  • Squirrel AI Learning, Seattle, Washington, USA
ORCID
Venue
Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
Publisher
ACM
Published
2025-08-03
Pages
6216-6226
DOI
10.1145/3711896.3736561
OpenAlex
W4412877234
Cited by
34
References
38
Citations by year
2026: 28 · 2025: 6
FWCI
113.78
Type
conference paper
Language
en
Open access
Locations
  • ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Sociology and Political Science › Access Control and Trust
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
LLM agents, multi-agent systems, security threats, countermeasures, agent taxonomy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review