CoChat
Sign Up
PreprintCited by 1Adversarial Robustness in Machine LearningSecurity and Verification in ComputingInformation and Cyber Security

The DOI matches its DataCite record; 17 of the 17 references it cites were found in a registry or an index.

A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework

Open access
Chu, Kexin·2026·arXiv
DOI 10.48550/arxiv.2604.23338arXiv 2604.23338
Open this paper in CoChat — free accountOpen PDF

Abstract

Agentic AI systems introduce a security surface that is qualitatively different from that of stateless LLMs. They persist memory, invoke external tools, coordinate with peer agents, and operate across sessions, allowing attacks to emerge not only at the prompt interface but also through architectural state, delegated authority, and long-horizon interactions. Existing security taxonomies, however, primarily organize threats by attack type, such as prompt injection or jailbreaking, and therefore obscure where in the agentic stack a threat arises and over what timescale it manifests. We propose the Layered Attack Surface Model (\lasm), a structural taxonomy for agentic AI security. \lasm decomposes the agentic stack into seven layers -- Foundation, Cognitive, Memory, Tool Execution, Multi-Agent Coordination, Ecosystem, and Governance -- and augments them with a four-class temporality axis covering instantaneous, session-persistent, cross-session cumulative, and sub-session-stack threats. We use this 7$\times$4 framework to analyze 116 papers from 2021--2026. The resulting map reveals that the upper layers of the agentic stack remain sharply under-explored, especially for long-horizon and stack-propagating threats; multiple documented attack regions have no corresponding defenses; and current benchmarks provide no coverage for cross-session or sub-session-stack failure modes. We further derive a cross-layer defense taxonomy, defense recipes for canonical attack classes, and a dependency DAG that separates near-term engineering gaps from fundamental research challenges. We release the per-paper coding, robustness scripts, and a reference Agent Bill of Materials schema to support reproducible analysis.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2604.23338

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 118

  1. Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian et al., "ToolLLM: Facilitating large language models to master 16000+ real-world APIs," in International Conference on Learning Representations (ICLR), 2024, arXiv:2307.16789. · arXiv
  2. S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, "Gorilla: Large language model connected with massive APIs," in Advances in Neural Information Processing Systems (NeurIPS), 2024, arXiv:2305.15334. · arXiv
  3. Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, "HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face," in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2303.17580. · arXiv
  4. G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, "Voyager: An Open-Ended Embodied Agent with Large Language Models," arXiv preprint arXiv:2305.16291, 2023. · arXiv
  5. L. Wang, C. Ma, X. Feng et al., "A Survey on Large Language Model Based Autonomous Agents," Frontiers of Computer Science, vol. 18, 2024.
  6. R. Nakano, J. Hilton et al., "WebGPT: Browser-assisted question- answering with human feedback," arXiv preprint arXiv:2112.09332, 2022. · arXiv
  7. Z. Niu, H. Liang, F. Zhang, and X. Li, "Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast," arXiv preprint arXiv:2402.08567, 2024. · arXiv
  8. X. Hou et al., "Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions," ACM Transactions on Software Engineering and Methodology, 2025, arXiv:2503.23278. · arXiv
  9. Z. Wang et al., "AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways," ACM Computing Surveys, vol. 57, no. 7, pp. 182:1-182:38, 2025.
  10. A. Chhabra, S. Datta, S. K. Nahin, and P. Mohapatra, "Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges," arXiv preprint arXiv:2510.23883, 2025. · arXiv
  11. Y. Tang, Y. Liu, J. Lan, Z. Yan, and E. Gelenbe, "Security of LLM-based Agents Regarding Attacks, Defenses, and Applications: A Comprehensive Survey," Information Fusion, vol. 127, 2026.
  12. M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow et al., "PRISMA 2020 Explanation and Elaboration: Updated Guidance and Exemplars for Reporting Systematic Reviews," BMJ, vol. 372, p. n160, 2021.
  13. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, "Not What You've Signed Up For: Compromising Real-World LLM- Integrated Applications with Indirect Prompt Injection," arXiv preprint arXiv:2302.12173, 2023. · arXiv
  14. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems (NeurIPS), 2017, arXiv:1706.03762. · arXiv
  15. OpenAI, "GPT-4 Technical Report," arXiv preprint arXiv:2303.08774, 2023. · arXiv
  16. H. Touvron, L. Martin et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
  17. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feedback," in NeurIPS, 2022.
  18. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon et al., "Constitutional AI: Harmlessness from AI Feedback," arXiv preprint arXiv:2212.08073, 2022. · arXiv
  19. R. Bommasani, D. A. Hudson et al., "On the opportunities and risks of foundation models," arXiv preprint arXiv:2108.07258, 2021. · arXiv
  20. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," in Advances in Neural Information Processing Systems (NeurIPS), 2022, neurIPS 2022.
  21. C. Lam, J. Li, L. Zhang, and K. Zhao, "Governing Evolving Memory in LLM Agents: Risks, Mechanisms, and the Stability and Safety Governed Memory (SSGM) Framework," arXiv preprint arXiv:2603.11768, 2025. · arXiv
  22. E. Hubinger et al., "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training," arXiv preprint arXiv:2401.05566, 2024. · arXiv
  23. N. Maloyan et al., "Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems," International Journal of Open Information Technologies, 2025, arXiv:2601.17548. · arXiv
  24. W. Zou, R. Geng, B. Wang, and J. Jia, "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models," arXiv preprint arXiv:2402.07867, 2024. · arXiv
  25. A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, "Universal and Transferable Adversarial Attacks on Aligned Language Models," arXiv preprint arXiv:2307.15043, 2023. · arXiv
  26. E. Hubinger, C. van Merwijk, V. Mikulik, J. Steen, and S. Garrabrant, "Risks from Learned Optimization in Advanced Machine Learning Systems," arXiv preprint arXiv:1906.01820, 2019. · arXiv
  27. X. Yang, X. Wang, Q. Zhang, L. Petzold, W. Y. Wang, X. Zhao, and D. Lin, "Shadow alignment: The ease of subverting safely-aligned language models," arXiv preprint arXiv:2310.02949, 2023. · arXiv
  28. A. Wan, E. Wallace, S. Shen, and D. Klein, "Poisoning language models during instruction tuning," in Proceedings of the 40th International Conference on Machine Learning (ICML), 2023, arXiv:2305.00944. · arXiv
  29. Z. Lin, C. Li, and K. Chen, "A Survey on the Security of Long- Term Memory in LLM Agents: Toward Mnemonic Sovereignty," arXiv preprint arXiv:2604.16548, 2025. · arXiv
  30. A. Pan, K. Bhatia, and J. Steinhardt, "The Effects of Reward Misspeci- fication: Mapping and Mitigating Misaligned Models," in International Conference on Learning Representations (ICLR), 2022, iCLR 2022; arXiv:2201.03544. · arXiv
  31. J. Zhu, D. Gandhi, H. Joshi, A. R. Mianroodi, S. A. Kocak, and D. Ra- machandran, "Interpreting Agentic Systems: Beyond Model Explana- tions to System-Level Accountability," arXiv preprint arXiv:2601.17168, 2025. · arXiv
  32. A. Bondarenko et al., "Demonstrating specification gaming in reasoning models," arXiv preprint arXiv:2502.13295, 2025. · arXiv
  33. Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, "AgentPoison: Red- teaming LLM agents via poisoning memory or knowledge bases," in Advances in Neural Information Processing Systems (NeurIPS), 2024, arXiv:2407.12784. · arXiv
  34. Y. Zhang, Q. Deng, T. Zhao, Z. Li, X. Su, Z. Zhu, H. Zhang, and C. Zhuge, "HijackRAG: Hijacking attacks against retrieval-augmented large language models," arXiv preprint arXiv:2410.22832, 2024. · arXiv
  35. J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger, "Defining and characterizing reward hacking," in Advances in Neural Information Processing Systems (NeurIPS), 2022, arXiv:2209.13085. · arXiv
  36. M. Nakamura, A. Kumar, S. Das, S. Abdelnabi, S. Mahmud, F. Fioretto, S. Zilberstein, and E. Bagdasarian, "Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems," arXiv preprint arXiv:2602.15198, 2025. · arXiv
  37. Y. Jo, M. Jeong, J. Im, and J. Kang, "Byzantine-robust decentralized coordination of LLM agents," arXiv preprint arXiv:2507.14928, 2025. · arXiv
  38. J. Weng, Y. Yao, H. Chen, Y. Xie, Z. Li, P. Hu, and D. Gu, "Large language model supply chain: A research agenda," arXiv preprint arXiv:2404.12736, 2024. · arXiv
  39. J. Yi, J. Ye, Z. Chen, R. Xiong, M. Xu, J. Li, L. Ma, T. Gui, Q. Zhang, and X. Huang, "Jailbreak and guard aligned language models with only few in-context demonstrations," arXiv preprint arXiv:2310.06387, 2023. · arXiv
  40. A. Wei, N. Haghtalab, and J. Steinhardt, "Jailbroken: How Does LLM Safety Training Fail?" in Advances in Neural Information Processing Systems (NeurIPS), 2023, neurIPS 2023.
  41. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, "Intriguing properties of neural networks," in International Conference on Learning Representations (ICLR), 2014, arXiv:1312.6199. · arXiv
  42. I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and Harnessing Adversarial Examples," arXiv preprint arXiv:1412.6572, 2014. · arXiv
  43. B. Biggio and F. Roli, "Wild patterns: Ten years after the rise of adversarial machine learning," Pattern Recognition, vol. 84, pp. 317- 331, 2018, arXiv:1712.03141. · arXiv
  44. X. Liu, N. Xu, M. Chen, and C. Xiao, "AutoDAN: Generating stealthy jailbreak prompts on aligned large language models," in International Conference on Learning Representations (ICLR), 2024, arXiv:2310.04451. · arXiv
  45. P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, "Jailbreaking black box large language models in twenty queries," in International Conference on Learning Representations (ICLR), 2024, arXiv:2310.08419. · arXiv
  46. C. Anil, E. Durmus, M. Sharma, J. Benton, E. Perez, R. Grosse, D. Duvenaud et al., "Many-Shot Jailbreaking," in Advances in Neural Information Processing Systems (NeurIPS), 2024, anthropic Technical Report; also presented at NeurIPS 2024.
  47. C. Wu et al., "Dissecting Adversarial Robustness of Multimodal LM Agents," in ICLR 2025, 2025, arXiv:2406.12814. · arXiv
  48. F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, "Stealing machine learning models via prediction APIs," in 25th USENIX Security Symposium (USENIX Security), 2016, arXiv:1609.02943. · arXiv
  49. R. Shokri, M. Stronati, C. Song, and V. Shmatikov, "Membership inference attacks against machine learning models," in IEEE Symposium on Security and Privacy (S&P), 2017.
  50. N. Carlini, D. Paleka, K. Dvijotham, T. Steinke, J. Hayase, A. F. Cooper, K. Lee, M. Jagielski, M. Nasr, A. Conmy et al., "Stealing part of a production language model," in International Conference on Machine Learning (ICML), 2024, arXiv:2403.06634. · arXiv
  51. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel, "Extracting training data from large language models," in 30th USENIX Security Symposium, 2021, pp. 2633-2650, arXiv:2012.07805. · arXiv
  52. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramèr, and C. Zhang, "Quantifying memorization across neural language models," in In- ternational Conference on Learning Representations (ICLR), 2023, arXiv:2202.07646. · arXiv
  53. M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee, "Scalable extraction of training data from (production) language models," arXiv preprint arXiv:2311.17035, 2023. · arXiv
  54. W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, "Detecting pretraining data from large language models," in International Conference on Learning Representations (ICLR), 2024, arXiv:2310.16789. · arXiv
  55. M. Fredrikson, S. Jha, and T. Ristenpart, "Model inversion attacks that exploit confidence information and basic countermeasures," in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS), 2015.
  56. F. Perez and I. Ribeiro, "Ignore Previous Prompt: Attack Techniques For Language Models," in EMNLP 2022 Workshop on Trustworthy NLP, 2022.
  57. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, "Direct Preference Optimization: Your language model is secretly a reward model," in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2305.18290. · arXiv
  58. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," in International Conference on Learning Representations (ICLR), 2018, arXiv:1706.06083. · arXiv
  59. J. M. Cohen, E. Rosenfeld, and J. Z. Kolter, "Certified adversarial robustness via randomized smoothing," in International Conference on Machine Learning (ICML), 2019, arXiv:1902.02918. · arXiv
  60. H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa, "Llama Guard: LLM-based input-output safeguard for human-AI conversations," arXiv preprint arXiv:2312.06674, 2023. · arXiv
  61. M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks, "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal," arXiv preprint arXiv:2402.04249, 2024. · arXiv
  62. K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, N. Z. Gong, Y. Zhang, and X. Xie, "PromptBench: Towards evaluating the robustness of large language models on adversarial prompts," arXiv preprint arXiv:2306.04528, 2023. · arXiv
  63. D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, A. Jones, S. Bowman et al., "Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned," arXiv preprint arXiv:2209.07858, 2022. · arXiv
  64. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, "Red teaming language models with language models," in Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), 2022, arXiv:2202.03286. · arXiv
  65. J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, "A watermark for large language models," in International Conference on Machine Learning (ICML), 2023, arXiv:2301.10226. · arXiv
  66. M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, "Deep learning with differential privacy," in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS), 2016.
  67. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, "Tree of Thoughts: Deliberate problem solving with large language models," in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2305.10601. · arXiv
  68. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang et al., "Self-refine: Iterative refinement with self-feedback," in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2303.17651. · arXiv
  69. N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, "Reflexion: Language agents with verbal reinforcement learning," in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2303.11366. · arXiv
  70. M. Turpin, J. Michael, E. Perez, and S. R. Bowman, "Language models don't always say what they think: Unfaithful explanations in chain- of-thought prompting," in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2305.04388. · arXiv
  71. Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, Q. Guo, M. Wang, and H. Wang, "Retrieval-augmented generation for large language models: A survey," arXiv preprint arXiv:2312.10997, 2023. · arXiv
  72. C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez, "MemGPT: Towards LLMs as operating systems," arXiv preprint arXiv:2310.08560, 2024. · arXiv
  73. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, "Not what you've signed up for: Compromising real-world LLM- integrated applications with indirect prompt injection," in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec), 2023, arXiv:2302.12173. · arXiv
  74. J. Liu, Y. Deng, W. Zheng, W. Yao, Y. Jiang, L. Lu, Y. Liu, M. Xue, and S. Zheng, "Prompt injection attacks against LLM-integrated applications," arXiv preprint arXiv:2306.05499, 2023. · arXiv
  75. Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, "Formalizing and benchmarking prompt injection attacks and defenses," in 33rd USENIX Security Symposium (USENIX Security), 2024, arXiv:2310.12815. · arXiv
  76. H. Zhang, J. Huang, K. Mei, Y. Yao, Z. Wang, C. Zhan, H. Wang, and Y. Zhang, "Agent security bench (ASB): Formalizing and benchmarking attacks and defenses in LLM-based agents," in International Conference on Learning Representations (ICLR), 2025, arXiv:2410.02644. · arXiv
  77. F. Wu, S. Wu, Y. Cao, and C. Xiao, "WIPI: A new web threat for LLM-driven web agents," arXiv preprint arXiv:2402.16965, 2024. · arXiv
  78. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig, "WebArena: A realistic web environment for building autonomous agents," in International Conference on Learning Representations (ICLR), 2024, arXiv:2307.13854. · arXiv
  79. S. Yao, H. Chen, J. Yang, and K. Narasimhan, "WebShop: Towards scalable real-world web interaction with grounded language agents," in Advances in Neural Information Processing Systems (NeurIPS), 2022, arXiv:2207.01206. · arXiv
  80. D. Kang et al., "LLM Agents can Autonomously Exploit One-day Vulnerabilities," arXiv preprint arXiv:2404.08144, 2024. · arXiv
  81. --, "Teams of LLM Agents can Exploit Zero-Day Vulnerabilities," arXiv preprint arXiv:2406.01637, 2024. · arXiv
  82. D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto, "Exploiting Novel GPT-4 APIs," arXiv preprint arXiv:2312.14328, 2023. · arXiv
  83. J. H. Saltzer and M. D. Schroeder, "The protection of information in computer systems," Proceedings of the IEEE, vol. 63, no. 9, pp. 1278-1308, 1975.
  84. N. Tran et al., "Least Privilege for AI Agents: Principles and Imple- mentation Strategies," arXiv preprint arXiv:2411.10452, 2024. · arXiv
  85. S. Chen, J. Piet, C. Sitawarin, and D. Wagner, "StruQ: Defending against prompt injection with structured queries," arXiv preprint arXiv:2402.06363, 2024. · arXiv
  86. S. Gulyamov, S. Gulyamov, A. Rodionov, R. Khursanov, K. Mekhmonov, D. Babaev, and A. Rakhimjonov, "Prompt Injection Attacks in Large Language Models and AI Agent Systems: A Comprehensive Review of Vulnerabilities, Attack Vectors, and Defense Mechanisms," Information, vol. 17, no. 1, p. 54, 2026.
  87. Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto, "Identifying the risks of LM agents with an LM-emulated sandbox," in International Conference on Learning Representations (ICLR), 2024, arXiv:2309.15817. · arXiv
  88. Q. Wu, G. Bansal, J. Zhang, Y. Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, "AutoGen: Enabling next-gen LLM applications via multi-agent conversation," arXiv preprint arXiv:2308.08155, 2023. · arXiv
  89. S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin et al., "MetaGPT: Meta programming for a multi-agent collaborative framework," in International Conference on Learning Representations (ICLR), 2024, arXiv:2308.00352. · arXiv
  90. C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun, "ChatDev: Communicative agents for software development," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024, arXiv:2307.07924. · arXiv
  91. G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, "CAMEL: Communicative agents for "mind" exploration of large language model society," in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2303.17760. · arXiv
  92. B. Qiao, L. Li, X. Zhang, S. He, Y. Kang, C. Zhang, F. Yang, H. Dong, J. Zhao, S. Qin et al., "TaskWeaver: A code-first agent framework," arXiv preprint arXiv:2311.17541, 2023. · arXiv
  93. B. Zhang, J. Tan, Y. Shen, A. Salem, M. Backes, S. Zannettou, and Y. Zhang, "Breaking agents: Compromising autonomous LLM agents through malfunction amplification," arXiv preprint arXiv:2407.20859, 2024. · arXiv
  94. Y. Zhang et al., "Evil Geniuses: Delving into the Safety of LLM-based Agents," arXiv preprint arXiv:2311.11855, 2024. · arXiv
  95. L. Lamport, R. Shostak, and M. Pease, "The Byzantine generals problem," ACM Transactions on Programming Languages and Systems, vol. 4, no. 3, pp. 382-401, 1982.
  96. M. Castro and B. Liskov, "Practical Byzantine fault tolerance," in 3rd USENIX Symposium on Operating Systems Design and Implementation (OSDI), 1999.
  97. T. Gu, B. Dolan-Gavitt, and S. Garg, "BadNets: Identifying vulnera- bilities in the machine learning model supply chain," arXiv preprint arXiv:1708.06733, 2017. · arXiv
  98. J. Xue, M. Zheng, T. Hua, Y. Shen, Y. Liu, L. Boloni, and Q. Lou, "TrojLLM: A black-box trojan prompt attack on large language models," in Advances in Neural Information Processing Systems (NeurIPS), 2023, arXiv:2306.06815. · arXiv
  99. X. Chen, C. Liu, B. Li, K. Lu, and D. Song, "Targeted backdoor attacks on deep learning systems using data poisoning," arXiv preprint arXiv:1712.05526, 2017. · arXiv
  100. Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, "Trojaning attack on neural networks," in 25th Annual Network and Distributed System Security Symposium (NDSS), 2018.
  101. X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, "Fine-tuning aligned language models compromises safety, even when users do not intend to!" in International Conference on Learning Representations (ICLR), 2024, arXiv:2310.03693. · arXiv
  102. H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, "Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions," in IEEE Symposium on Security and Privacy (S&P), 2022, arXiv:2108.09293. · arXiv
  103. R. Schuster, C. Song, E. Tromer, and V. Shmatikov, "You autocomplete me: Poisoning vulnerabilities in neural code completion," in 30th USENIX Security Symposium, 2021.
  104. H. Aghakhani, W. Dai, A. Manoel, X. Fernandes, A. Kharkar, C. Kruegel, G. Vigna, D. Evans, B. Zorn, and R. Sim, "TrojanPuzzle: Covertly poisoning code-suggestion models," in 2024 IEEE Symposium on Security and Privacy (SP), 2024, arXiv:2301.02344. · arXiv
  105. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, "Concrete problems in AI safety," arXiv preprint arXiv:1606.06565, 2016. · arXiv
  106. D. Hendrycks, M. Mazeika, and T. Woodside, "An Overview of Catastrophic AI Risks," arXiv preprint arXiv:2306.12001, 2023. · arXiv
  107. C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, L. Gao, L. Aschen- brenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, I. Sutskever, and J. Wu, "Weak-to-strong generalization: Eliciting strong capabilities with weak supervision," in International Conference on Machine Learning (ICML), 2024, arXiv:2312.09390. · arXiv
  108. I. Gabriel, "Artificial intelligence, values, and alignment," Minds and Machines, vol. 30, pp. 411-437, 2020.
  109. D. Hadfield-Menell, S. Russell, P. Abbeel, and A. Dragan, "Cooperative inverse reinforcement learning," in Advances in Neural Information Processing Systems (NIPS), 2016, arXiv:1606.03137. · arXiv
  110. European Parliament, "Regulation (EU) 2024/1689: Artificial Intelli- gence Act," 2024, official Journal of the European Union.
  111. The White House, "Executive order on the safe, secure, and trustworthy development and use of artificial intelligence," Executive Order 14110, Federal Register Vol. 88 No. 210, 2023.
  112. P. Liang, R. Bommasani et al., "Holistic evaluation of language models," Transactions on Machine Learning Research, 2022, arXiv:2211.09110. · arXiv
  113. S. Lin, J. Hilton, and O. Evans, "TruthfulQA: Measuring how models mimic human falsehoods," in Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (ACL), 2022, arXiv:2109.07958. · arXiv
  114. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, J. Tang, J. Yao et al., "AgentBench: Evaluating LLMs as Agents," arXiv preprint arXiv:2308.03688, 2023. · arXiv
  115. E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fischer, and M. Vechev, "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents," in Ad- vances in Neural Information Processing Systems (NeurIPS), 2024, arXiv:2406.13352. · arXiv
  116. S. Zhang et al., "AgentSecBench: Benchmarking Agent Security Against Prompt Injection and Tool Misuse," arXiv preprint arXiv:2410.02507, 2024. · arXiv
  117. J. Nöther, A. Singla, and G. Radanovic, "Benchmarking the Robustness of Agentic Systems to Adversarially-Induced Harms," arXiv preprint arXiv:2508.16481, 2025. · arXiv
  118. N. Papernot, P. McDaniel, A. Sinha, and M. P. Wellman, "SoK: Security and privacy in machine learning," in 3rd IEEE European Symposium on Security and Privacy (EuroS&P), 2018.

Cited by

Sorted by citations

See all 1

Metadata

Authors
Chu, K.
Corresponding author
Chu, Kexin
ORCID
Chu, Kexin
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2604.23338
arXiv
2604.23338
OpenAlex
W7157103096
Cited by
1
References
17
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
Keywords
agentic AI security, prompt injection, jailbreaking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review