The DOI matches its Crossref record; 131 of the 131 references it cites were found in a registry or an index.
AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways
Open accessPublished version of a preprint
Abstract
An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. AI agents, capable of perceiving user inputs, reasoning and planning tasks, and executing actions, have seen remarkable advancements in algorithm development and task performance. However, the security challenges they pose remain under-explored and unresolved. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps: unpredictability of multi-step user inputs, complexity in internal executions, variability of operational environments, and interactions with untrusted external entities. By systematically reviewing these threats, this article highlights both the progress made and the existing limitations in safeguarding AI agents. The insights provided aim to inspire further research into addressing the security threats associated with AI agents, thereby fostering the development of more robust and secure AI agent applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI Agents are typically designed as single-entity systems that perform goal-directed tasks by utilizing external tools, applying sequential reasoning, and integrating real-time information to complete well-defined functions [25,43].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, while pre-modern expert systems required manual updates to static knowledge bases, modern agents leverage emergent neural architectures to generalize across tasks [25].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI Agents are typically designed as single-entity systems that perform goal-directed tasks by utilizing external tools, applying sequential reasoning, and integrating real-time information to complete well-defined functions [24], [45].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [24]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“AI Agents often include basic mechanisms for interacting with dynamic inputs, allowing them to respond to real-time stimuli such as user requests, external API calls, or state changes in software environments [25,71].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Already, expanded attack surfaces and emergent risky behaviors have been observed in autonomous agent experiments [49,50,51].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [49]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“Due to the associative abilities of LLMs [25], frequent occurrences of pronouns and identity markers, such as gender, race, nationality, and culture in training data, can bias AI agents in processing data [59, 162].”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“Additionally, the other effective measures can be applied, such as fine-tuning AI agents specifically for indirect prompt injection [196, 204], alignment [121], and employing methods such as prompt engineering and post-training classifier-based security approaches [68].”
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 1×
“This limitation restricts their ability to retain and use extensive context from previous interactions.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Research works printed but not linked: 114
- Mahyar Abbasian Iman Azimi Amir M. Rahmani and Ramesh Jain. 2023. Conversational health agents: A personalized LLM-powered agent framework. Retrieved from https://arxiv.org/abs/2310.02374 · arXiv
- Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. GPT-4 technical report. Retrieved from https://arxiv.org/pdf/2303.08774 · arXiv
- Divyansh Agarwal Alexander R. Fabbri Philippe Laban Shafiq Joty Caiming Xiong and Chien-Sheng Wu. 2024. Investigating the prompt leakage effect and black-box defenses for multi-turn LLM interactions. Retrieved from https://arxiv.org/pdf/2404.16251v1 · arXiv
- Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. Retrieved from https://arxiv.org/pdf/2308.14132 · arXiv
- Anthropic. 2024. Many-shot Jailbreaking. Retrieved from https://www.anthropic.com/research/many-shot-jailbreaking
- Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen Nova DasSarma Dawn Drain Stanislav Fort Deep Ganguli Tom Henighan et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. Retrieved from https://arxiv.org/pdf/2204.05862 · arXiv
- Rishabh Bhardwaj and Soujanya Poria. 2023. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. Retrieved from https://arxiv.org/pdf/2310.14303 · arXiv
- Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive environments with online reinforcement learning. In Proceedings of the ICML.
- Dake Chen Hanbin Wang Yunhao Huo Yuzhao Li and Haoyang Zhang. 2023. Gamegpt: Multi-agent collaborative framework for game development. Retrieved from https://arxiv.org/pdf/2310.08067 · arXiv
- Xuan Chen Yuzhou Nie Lu Yan Yunshu Mao Wenbo Guo and Xiangyu Zhang. 2024. RL-JACK: Reinforcement learning-powered black-box jailbreaking attack against LLMs. Retrieved from https://arxiv.org/pdf/2406.08725 · arXiv
- Yongchao Chen Jacob Arkin Yang Zhang Nicholas Roy and Chuchu Fan. 2023. Scalable multi-robot collaboration with large language models: Centralized or decentralized systems? Retrieved from https://arxiv.org/html/2309.15943v2
- Zhaorun Chen Zhen Xiang Chaowei Xiao Dawn Song and Bo Li. 2024. Agentpoison: Red-teaming LLM agents via poisoning memory or knowledge bases. Retrieved from https://arxiv.org/pdf/2407.12784 · arXiv
- Steffi Chern Zhen Fan and Andy Liu. 2024. Combating adversarial attacks with multi-agent debate. Retrieved from https://arxiv.org/pdf/2401.05998 · arXiv
- Stav Cohen Ron Bitton and Ben Nassi. 2024. Here comes the AI worm: Unleashing zero-click worms that target GenAI-powered applications. Retrieved from https://arxiv.org/pdf/2403.02817 · arXiv
- Maxwell Crouse Ibrahim Abdelaziz Kinjal Basu Soham Dan Sadhana Kumaravel Achille Fokoue Pavan Kapanipathi and Luis Lastras. 2023. Formally specifying the high-level behavior of LLM-based agents. Retrieved from https://arxiv.org/pdf/2310.08535 · arXiv
- Chenhang Cui Gelei Deng An Zhang Jingnan Zheng Yicong Li Lianli Gao Tianwei Zhang and Tat-Seng Chua. 2024. Safe+ Safe= Unsafe? Exploring how safe images can be exploited to jailbreak large vision-language models. Retrieved from https://arxiv.org/pdf/2411.11496 · arXiv
- Tianyu Cui Yanling Wang Chuanpu Fu Yong Xiao Sijia Li Xinhao Deng Yunpeng Liu Qinglin Zhang Ziyi Qiu Peiyang Li et al. 2024. Risk taxonomy mitigation and assessment benchmarks of large language model systems. Retrieved from https://arxiv.org/pdf/2401.05778 · arXiv
- Yiran Ding Li Lyna Zhang Chengruidong Zhang Yuanyuan Xu Ning Shang Jiahang Xu Fan Yang and Mao Yang. 2024. LongRoPE: Extending LLM context window beyond 2 million tokens. Retrieved from https://arxiv.org/pdf/2402.13753 · arXiv
- Tian Dong Guoxing Chen Shaofeng Li Minhui Xue Rayne Holland Yan Meng Zhen Liu and Haojin Zhu. 2023. Unleashing cheapfakes through trojan plugins of large language models. Retrieved from https://arxiv.org/pdf/2312.00374v1 · arXiv
- Yingkai Dong Zheng Li Xiangtao Meng Ning Yu and Shanqing Guo. 2024. Jailbreaking text-to-image models with LLM-based agents. Retrieved from https://arxiv.org/pdf/2408.00523 · arXiv
- Yilun Du Shuang Li Antonio Torralba Joshua B. Tenenbaum and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. Retrieved from https://arxiv.org/pdf/2305.14325 · arXiv
- Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. 2023. Guiding pretraining in reinforcement learning with large language models. In Proceedings of the ICML. PMLR.
- Eva Eigner and Thorsten Händler. 2024. Determinants of LLM-assisted decision-making. Retrieved from https://arxiv.org/pdf/2402.17385 · arXiv
- Aysan Esmradi, Daniel Wankit Yip, and Chun Fai Chan. 2023. A comprehensive survey of attack techniques, implementation, and mitigation strategies in large language models. In Proceedings of the UbiSec.
- Isabel O. Gallegos Ryan A. Rossi Joe Barrow Md Mehrab Tanjim Sungchul Kim Franck Dernoncourt Tong Yu Ruiyi Zhang and Nesreen K. Ahmed. 2023. Bias and fairness in large language models: A survey. Retrieved from https://arxiv.org/pdf/2309.00770 · arXiv
- Jonas Geiping Alex Stein Manli Shu Khalid Saifullah Yuxin Wen and Tom Goldstein. 2024. Coercing LLMs to do and reveal (almost) anything. Retrieved from https://arxiv.org/abs/2402.14020 · arXiv
- Kai Greshake Sahar Abdelnabi Shailesh Mishra Christoph Endres Thorsten Holz and Mario Fritz. 2023. More than you’ve asked for: A comprehensive analysis of novel prompt injection threats to application-integrated large language models. Retrieved from https://arxiv.org/pdf/2302.12173 · arXiv
- Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In Proceedings of the AISec. 79–90.
- Xiangming Gu Xiaosen Zheng Tianyu Pang Chao Du Qian Liu Ye Wang Jing Jiang and Min Lin. 2024. Agent smith: A single image can jailbreak one million multimodal LLM agents exponentially fast. Retrieved from https://arxiv.org/pdf/2402.08567 · arXiv
- Michael Guastalla, Yiyi Li, Arvin Hekmati, and Bhaskar Krishnamachari. 2023. Application of large language models to DDoS attack detection. In Proceedings of the SmartSP.
- Rui Hao Linmei Hu Weijian Qi Qingliu Wu Yirui Zhang and Liqiang Nie. 2023. ChatLLM network: More brains more intelligence. Retrieved from https://arxiv.org/pdf/2304.12998 · arXiv
- Kostas Hatalis, Despina Christou, Joshua Myers, Steven Jones, Keith Lambert, Adam Amos-Binks, Zohreh Dannenhauer, and Dustin Dannenhauer. 2023. Memory matters: The need to improve long-term memory in LLM-Agents. In Proceedings of the AAAI.
- Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In Proceedings of the ICLR.
- Yuki Hou, Haruki Tamoto, and Homei Miyashita. 2024. “My agent understands me better”: Integrating dynamic human-like memory recall and consolidation in LLM-based agents. In Proceedings of the CHI.
- Bin Hu Chenyang Zhao Pu Zhang Zihao Zhou Yuanhang Yang Zenglin Xu and Bin Liu. 2023. Enabling intelligent interactions between an agent and an LLM: A reinforcement learning approach. Retrieved from https://arxiv.org/pdf/2306.03604 · arXiv
- Xijie Huang Li Lyna Zhang Kwang-Ting Cheng and Mao Yang. 2023. Boosting LLM reasoning: Push the limits of few-shot learning with reinforced in-context pruning. Retrieved from https://arxiv.org/pdf/2312.08901 · arXiv
- Yue Huang Qihui Zhang Lichao Sun et al. 2023. Trustgpt: A benchmark for trustworthy and responsible large language models. Retrieved from https://arxiv.org/pdf/2306.11507 · arXiv
- S. Humeau, K. Shuster, M. Lachaux, and J. Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv. In Proceedings of the ICLR.
- Neel Jain Avi Schwarzschild Yuxin Wen Gowthami Somepalli John Kirchenbauer Ping-yeh Chiang Micah Goldblum Aniruddha Saha Jonas Geiping and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. Retrieved from https://arxiv.org/pdf/2309.00614 · arXiv
- Zhenlan Ji Daoyuan Wu Pingchuan Ma Zongjie Li and Shuai Wang. 2024. Testing and understanding erroneous planning in LLM agents through synthesized user inputs. Retrieved from https://arxiv.org/pdf/2404.17833 · arXiv
- Mintong Kang Nezihe Merve Gürel Ning Yu Dawn Song and Bo Li. 2024. C-RAG: Certified generation risks for retrieval-augmented language models. Retrieved from https://arxiv.org/pdf/2402.03181 · arXiv
- Aounon Kumar Chirag Agarwal Suraj Srinivas Soheil Feizi and Hima Lakkaraju. 2024. Certifying LLM safety against adversarial prompting. Retrieved from https://arxiv.org/pdf/2309.02705 · arXiv
- Patrick Levi and Christoph P. Neumann. 2024. Vocabulary attack to hijack large language model applications. Retrieved from https://arxiv.org/pdf/2404.02637 · arXiv
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the NeurIPS.
- Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative agents for ”Mind” exploration of large language model society. In Proceedings of the NeurIPS.
- Yuanchun Li Hao Wen Weijun Wang Xiangyu Li Yizhen Yuan Guohong Liu Jiacheng Liu Wenxing Xu Xiang Wang Yi Sun et al. 2024. Personal LLM agents: Insights and survey about the capability efficiency and security. Retrieved from https://arxiv.org/pdf/2401.05459 · arXiv
- Zelong Li Wenyue Hua Hao Wang He Zhu and Yongfeng Zhang. 2024. Formal-LLM: Integrating formal language and natural language for controllable LLM-based agents. Retrieved from https://arxiv.org/pdf/2402.00798 · arXiv
- Zeyi Liao Lingbo Mo Chejian Xu Mintong Kang Jiawei Zhang Chaowei Xiao Yuan Tian Bo Li and Huan Sun. 2024. Eia: Environmental injection attack on generalist web agents for privacy leakage. Retrieved from https://arxiv.org/pdf/2409.11295 · arXiv
- Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023. AvalonBench: Evaluating LLMs playing the game of avalon. In Proceedings of the NeurIPS Workshop.
- Baihan Lin Djallel Bouneffouf Guillermo Cecchi and Kush R. Varshney. 2023. Towards healthy AI: large language models need therapists too. Retrieved from https://arxiv.org/pdf/2304.00416 · arXiv
- Jiaju Lin Haoran Zhao Aochi Zhang Yiting Wu Huqiuyue Ping and Qin Chen. 2023. Agentsims: An open-source sandbox for large language model evaluation. Retrieved from https://arxiv.org/pdf/2308.04026 · arXiv
- Sheng Liu Lei Xing and James Zou. 2023. In-context vectors: Making in context learning more effective and controllable through latent space steering. Retrieved from https://arxiv.org/pdf/2311.06668 · arXiv
- Tong Liu Zizhuang Deng Guozhu Meng Yuekang Li and Kai Chen. 2023. Demystifying rce vulnerabilities in LLM-integrated apps. Retrieved from https://arxiv.org/pdf/2309.02926 · arXiv
- Yi Liu Gelei Deng Yuekang Li Kailong Wang Tianwei Zhang Yepang Liu Haoyu Wang Yan Zheng and Yang Liu. 2023. Prompt injection attack against LLM-integrated applications. Retrieved from https://arxiv.org/pdf/2306.05499 · arXiv
- Yi Liu Gelei Deng Zhengzi Xu Yuekang Li Yaowen Zheng Ying Zhang Lida Zhao Tianwei Zhang and Yang Liu. 2023. Jailbreaking chatgpt via prompt engineering: An empirical study. Retrieved from https://arxiv.org/pdf/2305.13860 · arXiv
- Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In Proceedings of the USENIX Security.
- Yang Liu Yuanshun Yao Jean-Francois Ton Xiaoying Zhang Ruocheng Guo Hao Cheng Yegor Klochkov Muhammad Faaiz Taufiq and Hang Li. 2023. Trustworthy LLMs: A survey and guideline for evaluating large language models’ alignment. Retrieved from https://arxiv.org/pdf/2308.05374 · arXiv
- Qinghua Lu Liming Zhu Xiwei Xu Zhenchang Xing Stefan Harrer and Jon Whittle. 2023. Building the future of responsible AI: A reference architecture for designing large language model based agents. Retrieved from https://arxiv.org/pdf/2311.13148 · arXiv
- Nikhil Mehta, Milagro Teruel, Xin Deng, Sergio Figueroa Sanz, Ahmed Awadallah, and Julia Kiseleva. 2024. Improving grounded language understanding in a collaborative environment by interacting with agents through help feedback. In Proceedings of the EACL.
- Kai Mei Zelong Li Shuyuan Xu Ruosong Ye Yingqiang Ge and Yongfeng Zhang. 2024. AIOS: LLM agent operating system. Retrieved from https://arxiv.org/pdf/2403.16971 · arXiv
- Lingbo Mo Zeyi Liao Boyuan Zheng Yu Su Chaowei Xiao and Huan Sun. 2024. A trembling house of cards? Mapping adversarial attacks against language agents. Retrieved from https://arxiv.org/pdf/2402.10196 · arXiv
- Sumeet Ramesh Motwani, Mikhail Baranchuk, Lewis Hammond, and Christian Schroeder de Witt. 2023. A perfect collusion benchmark: How can AI agents be prevented from colluding with information-theoretic undetectability?. In Proceedings of the NeurIPS Workshop.
- Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. 2024. Generating benchmarks for factuality evaluation of language models. In Proceedings of the EACL.
- Tai Nguyen and Eric Wong. 2023. In-context example selection with influences. Retrieved from https://arxiv.org/pdf/2302.11042 · arXiv
- Aidan O’Gara. 2023. Hoodwinked: Deception and cooperation in a text-based game for language models. Retrieved from https://arxiv.org/pdf/2308.01404 · arXiv
- Yikang Pan Liangming Pan Wenhu Chen Preslav Nakov Min-Yen Kan and William Yang Wang. 2023. On the risk of misinformation pollution with large language models. Retrieved from https://arxiv.org/pdf/2305.13661 · arXiv
- Rodrigo Pedro Daniel Castro Paulo Carreira and Nuno Santos. 2023. From prompt injections to SQL injection attacks: How protected is your LLM-integrated web application? Retrieved from https://arxiv.org/pdf/2308.01990 · arXiv
- Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. Retrieved from https://arXiv:2211.09527
- Steve Phelps and Rebecca Ranson. 2023. Of Models and Tin Men–a behavioural economics study of principal-agent problems in AI alignment using large-language models. Retrieved from https://arxiv.org/pdf/2307.11137 · arXiv
- Lukas Pöhler Valentin Schrader Alexander Ladwein and Florian von Keller. 2024. A technological perspective on misuse of available AI. Retrieved from https://arxiv.org/pdf/2403.15325 · arXiv
- Jack W. Rae Sebastian Borgeaud Trevor Cai Katie Millican Jordan Hoffmann Francis Song John Aslanides Sarah Henderson Roman Ring Susannah Young et al. 2021. Scaling language models: Methods analysis & insights from training Gopher. Retrieved from https://arxiv.org/pdf/2112.11446 · arXiv
- Fathima Abdul Rahman and Guang Lu. 2023. A contextualized real-time multimodal emotion recognition for conversational agents using graph convolutional networks in reinforcement learning. Retrieved from https://arxiv.org/pdf/2310.18363 · arXiv
- Alexander Robey Zachary Ravichandran Vijay Kumar Hamed Hassani and George J Pappas. 2024. Jailbreaking LLM-controlled robots. Retrieved from https://arxiv.org/pdf/2410.13691 · arXiv
- Sippo Rossi Alisia Marianne Michel Raghava Rao Mukkamala and Jason Bennett Thatcher. 2024. An early categorization of prompt injection attacks on large language models. Retrieved from https://arxiv.org/pdf/2402.00898 · arXiv
- Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the risks of LM agents with an LM-emulated sandbox. In Proceedings of the ICLR.
- Ahmed Salem Andrew Paverd and Boris Köpf. 2023. Maatphor: Automated variant analysis for prompt injection attacks. Retrieved from https://arxiv.org/pdf/2312.11513 · arXiv
- Leo Schwinn David Dobre Stephan Günnemann and Gauthier Gidel. 2023. Adversarial attacks and defenses in large language models: Old and new threats. Retrieved from https://arxiv.org/pdf/2310.19737 · arXiv
- Mrinank Sharma Meg Tong Tomasz Korbak David Duvenaud Amanda Askell Samuel R. Bowman Newton Cheng Esin Durmus Zac Hatfield-Dodds Scott R. Johnston et al. 2023. Towards understanding sycophancy in language models. Retrieved from https://arxiv.org/pdf/2310.13548 · arXiv
- Xinyue Shen Yixin Wu Michael Backes and Yang Zhang. 2024. Voice jailbreak attacks against GPT-4o. Retrieved from https://arxiv.org/abs/2405.19103 · arXiv
- Emily H. Soice Rafael Rocha Kimberlee Cordova Michael Specter and Kevin M. Esvelt. 2023. Can large language models democratize access to dual-use biotechnology? Retrieved from https://arxiv.org/pdf/2306.03809 · arXiv
- Yuxiang Sun Checheng Yu Junjie Zhao Wei Wang and Xianzhong Zhou. 2023. Self generated wargame AI: Double layer agent task planning based on large language model. Retrieved from https://arxiv.org/pdf/2312.01090 · arXiv
- Harini Suresh and John V. Guttag. 2019. A framework for understanding unintended consequences of machine learning. Retrieved from https://arxiv.org/pdf/1901.10002 · arXiv
- Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. 2024. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In Proceedings of the ICLR.
- Xiangru Tang Qiao Jin Kunlun Zhu Tongxin Yuan Yichi Zhang Wangchunshu Zhou Meng Qu Yilun Zhao Jian Tang Zhuosheng Zhang et al. 2024. Prioritizing safeguarding over autonomy: Risks of LLM agents for science. Retrieved from https://arxiv.org/pdf/2402.04247 · arXiv
- Mikhail Terekhov, Romain Graux, Eduardo Neville, Denis Rosset, and Gabin Kolly. 2023. Second-order Jailbreaks: Generative agents successfully manipulate through an intermediary. In Proceedings of the Multi-Agent Security Workshop @ NeurIPS’23. Retrieved from https://openreview.net/forum?id=HPmhaOTseN
- Yu Tian Xiao Yang Jingyuan Zhang Yinpeng Dong and Hang Su. 2023. Evil geniuses: Delving into the safety of llm-based agents. Retrieved from https://arxiv.org/pdf/2311.11855 · arXiv
- Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. 2024. Tensor trust: Interpretable prompt injection attacks from an online game. In Proceedings of the ICLR.
- Celine Wald and Lukas Pfahler. 2023. Exposing bias in online communities through large-scale language models. Retrieved from https://arxiv.org/pdf/2306.02294 · arXiv
- Eric Wallace Kai Xiao Reimar Leike Lilian Weng Johannes Heidecke and Alex Beutel. 2024. The instruction hierarchy: Training LLMs to prioritize privileged instructions. Retrieved from https://arxiv.org/pdf/2404.13208 · arXiv
- Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In Proceedings of the ICML.
- Fengxiang Wang Ranjie Duan Peng Xiao Xiaojun Jia YueFeng Chen Chongwen Wang Jialing Tao Hang Su Jun Zhu and Hui Xue. 2024. MRJ-Agent: An effective jailbreak agent for multi-round dialogue. Retrieved from https://arxiv.org/pdf/2411.03814 · arXiv
- Jerry Wei Chengrun Yang Xinying Song Yifeng Lu Nathan Hu Dustin Tran Daiyi Peng Ruibo Liu Da Huang Cosmo Du et al. 2024. Long-form factuality in large language models. Retrieved from https://arxiv.org/pdf/2403.18802 · arXiv
- Zeming Wei Yifei Wang and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. Retrieved from https://arxiv.org/pdf/2310.06387 · arXiv
- Roy Weiss Daniel Ayzenshteyn Guy Amit and Yisroel Mirsky. 2024. What was your prompt? A remote keylogging attack on AI assistants. Retrieved from https://arxiv.org/pdf/2403.09751 · arXiv
- Yotam Wolf Noam Wies Yoav Levine and Amnon Shashua. 2023. Fundamental limitations of alignment in large language models. Retrieved from https://arxiv.org/pdf/2304.11082 · arXiv
- Chen Henry Wu Jing Yu Koh Ruslan Salakhutdinov Daniel Fried and Aditi Raghunathan. 2024. Adversarial attacks on multimodal agents. Retrieved from https://arxiv.org/pdf/2406.12814v1 · arXiv
- Fangzhou Wu Shutong Wu Yulong Cao and Chaowei Xiao. 2024. WIPI: A new web threat for LLM-driven web agents. Retrieved from https://arxiv.org/pdf/2402.16965 · arXiv
- Junlin Xie Zhihong Chen Ruifei Zhang Xiang Wan and Guanbin Li. 2024. Large multimodal agents: A survey. Retrieved from https://arxiv.org/pdf/2402.15116 · arXiv
- Frank Xing. 2024. Designing heterogeneous LLM agents for financial sentiment analysis. Retrieved from https://arxiv.org/pdf/2401.05799 · arXiv
- Binfeng Xu Zhiyuan Peng Bowen Lei Subhabrata Mukherjee Yuchen Liu and Dongkuan Xu. 2023. Rewoo: Decoupling reasoning from observations for efficient augmented language models. Retrieved from https://arxiv.org/pdf/2305.18323 · arXiv
- Yuzhuang Xu Shuo Wang Peng Li Fuwen Luo Xiaolong Wang Weidong Liu and Yang Liu. 2023. Exploring large language models for communication games: An empirical study on werewolf. Retrieved from https://arxiv.org/pdf/2309.04658 · arXiv
- Zelai Xu Chao Yu Fei Fang Yu Wang and Yi Wu. 2023. Language agents with reinforcement learning for strategic play in the werewolf game. Retrieved from https://arxiv.org/pdf/2309.04658 · arXiv
- Xue Yan Yan Song Xinyu Cui Filippos Christianos Haifeng Zhang David Henry Mguni and Jun Wang. 2023. Ask more know better: Reinforce-learned prompt questions for decision making with large language models. Retrieved from https://arxiv.org/pdf/2310.18127 · arXiv
- Yong Yang Xuhong Zhang Yi Jiang Xi Chen Haoyu Wang Shouling Ji and Zonghui Wang. 2024. PRSA: Prompt reverse stealing attacks against large language models. Retrieved from https://arxiv.org/pdf/2402.19200 · arXiv
- Jingwei Yi Yueqi Xie Bin Zhu Keegan Hines Emre Kiciman Guangzhong Sun Xing Xie and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. Retrieved from https://arxiv.org/pdf/2312.14197 · arXiv
- Jiahao Yu Xingwei Lin and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. Retrieved from https://arxiv.org/pdf/2309.10253 · arXiv
- Zhiyuan Yu Xiaogeng Liu Shunning Liang Zach Cameron Chaowei Xiao and Ning Zhang. 2024. Don’t listen to me: Understanding and exploring jailbreak prompts of large language models. Retrieved from https://arxiv.org/pdf/2403.17336 · arXiv
- Yifan Zeng Yiran Wu Xiao Zhang Huazheng Wang and Qingyun Wu. 2024. AutoDefense: Multi-agent LLM defense against jailbreak attacks. Retrieved from https://arxiv.org/pdf/2403.04783 · arXiv
- Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua Tenenbaum, Tianmin Shu, and Chuang Gan. 2023. Building cooperative embodied agents modularly with large language models. In Proceedings of the NeurIPS Workshop.
- Wanpeng Zhang and Zongqing Lu. 2023. Rladapter: Bridging large language models to reinforcement learning in open worlds. Retrieved from https://arxiv.org/pdf/2309.17176 · arXiv
- Yuqi Zhang Liang Ding Lefei Zhang and Dacheng Tao. 2024. Intention analysis prompting makes large language models a good jailbreak defender. Retrieved from https://arxiv.org/pdf/2401.06561 · arXiv
- Zhexin Zhang Junxiao Yang Pei Ke and Minlie Huang. 2023. Defending large language models against jailbreaking attacks through goal prioritization. Retrieved from https://arxiv.org/pdf/2311.09096 · arXiv
- Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2024. Analyzing and mitigating object hallucination in large vision-language models. In Proceedings of the ICLR.
- Wei Zou Runpeng Geng Binghui Wang and Jinyuan Jia. 2024. PoisonedRAG: Knowledge poisoning attacks to retrieval-augmented generation of large language models. Retrieved from https://arxiv.org/pdf/2402.07867 · arXiv
Cited by
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“For instance, while pre-modern expert systems required manual updates to static knowledge bases, modern agents leverage emergent neural architectures to generalize across tasks [25].”
InfluentialAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
“For instance, while pre-modern expert systems required manual updates to static knowledge bases, modern agents leverage emergent neural architectures to generalize across tasks [25].”
InfluentialAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Exploring DeepSeek: A Survey on Advances, Applications, Challenges and Future Directions
Deng, Z., Ma, W., Han, Q., Zhou, W., Zhu, X., Wen, S., & Xiang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125498 · Cited by 224
Self-citationModel Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions
Hou, X., Zhao, Y., Wang, S., & Wang, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796519 · Cited by 167
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 108
Agentic AI: A Comprehensive Survey of Technologies, Applications, and Societal Implications
Pati, A. K. (2025) · IEEE Access · doi:10.1109/access.2025.3585609 · Cited by 100
“Methodologies, Technologies, and working Although a few works discuss methodologies and technologies used for designing these systems [7], [11], [14], [90], [91], several others ignored these [8], [9], [13], [15] methodological foundations.”
InfluentialOn protecting the data privacy of Large Language Models (LLMs) and LLM agents: A literature review
Yan, B., Li, K., Xu, M., Dong, Y., Zhang, Y., Ren, Z., & Cheng, X. (2025) · High-Confidence Computing · doi:10.1016/j.hcc.2025.100300 · Cited by 160
Tasmurzayev, N., Amangeldy, B., Иманбек, Б., Baigarayeva, Z., Imankulov, T., Dikhanbayeva, G., Amangeldi, I., & Sharipova, S. (2025) · Sensors · doi:10.3390/s25175272 · Cited by 67
“…for building AI agents, encompassing logic-oriented meth-ods, agent-oriented programming languages, cognitive architectures, emerging paradigms based on large language models (LLMs), multi-agent communication frameworks, simulation environments, and integrated intelligent platforms [116].”
Agentic AI: The age of reasoning—A review
Nisa, U., Shirazi, M., Saip, M. A., & Pozi, M. S. M. (2025) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2025.08.003 · Cited by 136
From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
Ferrag, M. A., Tihanyi, N., Hamouda, D., Maglaras, L., Lakas, A., & Debbah, M. (2026) · ICT Express · doi:10.1016/j.icte.2025.12.001 · Cited by 46
“Moreover, the absence of a shared discovery mechanism or common vocabulary for tool capabilities increases the risk of security gaps when onboarding new services or updating existing ones [7], [8], [9], [10].”
InfluentialAI Agents and Agentic Systems: Redefining Global it Management
Hughes, L., Dwivedi, Y. K., Li, K., Appanderanda, M., Albashrawi, M. A., & Chae, I. (2025) · Journal of Global Information Technology Management · doi:10.1080/1097198x.2025.2524286 · Cited by 46
“Hallucination is another critical challenge, particularly in multi-agent systems (Deng et al., 2025).”
Enhancing Safety in Autonomous Maritime Transportation Systems with Real-Time AI Agents
Durlik, I., Miller, T. I., Kostecka, E., Kozlovska, P., & Ślączka, W. (2025) · Applied Sciences · doi:10.3390/app15094986 · Cited by 45
Generative AI cybersecurity and resilience
Radanliev, P., Santos, O., & Ani, U. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1568360 · Cited by 47
“…IEEE, 2023; Roberts et al., 2021; Mökander et al., 2024), security (He et al., 2024; Porambage et al., 2019; Sarker et al., 2021; Mishra, 2023; Deng et al., 2024), and privacy (Bartoletti, 2019) implications accompanying its widespread deployment (Jobin et al., 2019; European Commission,…”
Deng, Z., Sun, R., Xue, M., Ma, W., Wen, S., Nepal, S., & Xiang, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3581103 · Cited by 36
“The use of LLMs [4], [5], [6], [7], [8] in critical infrastructure o ff ers significant benefits but also presents potential risks.”
Self-citationCyberRAG: An agentic RAG cyber attack classification and reporting tool
Blefari, F., Cosentino, C., Pironti, F. A., Furfaro, A., & Marozzo, F. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.108186 · Cited by 42
“These agents are designed to autonomously perform tasks such as continuous monitoring, anomaly detection, and threat mitigation, increasingly becoming a key component of modern security operations [3, 4].”
A roadmap for evaluating moral competence in large language models
Haas, J., Bridgers, S., Manzini, A., Henke, B., May, J., Levine, S., Weidinger, L., Shanahan, M., Lum, K., Gabriel, I., et al. (2026) · Nature · doi:10.1038/s41586-025-10021-1 · Cited by 25
Alabbadi, A., & Bajaber, F. (2025) · Sensors · doi:10.3390/s25123693 · Cited by 25
“Alternative to the above studies, researchers like Deng et al. [23,24] have discussed the concerns of security and challenges for the AI agents.”
A Survey on Agent Workflow – Status and Future
Yu, C., Cheng, Z., Cui, H., Gao, Y., Luo, Z., Wang, Y., Zheng, H., & Zhao, Y. (2025) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd64986.2025.11082076 · Cited by 69
“In cross-server attacks [41], attackers can execute attacks by using shadowing tool descriptions.”
Ray, P. P. (2026) · Expert Systems · doi:10.1111/exsy.70213 · Cited by 44
He, Y., Wang, E., Rong, Y., Cheng, Z., & Chen, H. (2025) · 2025 IEEE/ACM International Workshop on Responsible AI Engineering (RAIE) · doi:10.1109/raie66699.2025.00013 · Cited by 99
“The threats to LLMs and AI agents are different [49].”
Security of LLM-based agents regarding attacks, defenses, and applications: A comprehensive survey
Tang, Y., Liu, Y., Lan, J., Yan, Z., & Erol Gelenbe, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103941 · Cited by 43
Agentic AI Framework to Automate Traditional Farming for Smart Agriculture
Murad, M., Ahmed, M., din, N. U., Shahid, M. F., Siddiqui, S., Byers, D., Tanveer, M. H., & Voicu, R. C. (2026) · AgriEngineering · doi:10.3390/agriengineering8010008 · Cited by 21
From threat to trust: assessing security risks of agentic AI systems
Leo, M., Tan, F., Miao, T., & Anand, G. (2026) · International Journal of Information Security · doi:10.1007/s10207-025-01185-y · Cited by 25
“1 Security risks within agentic AI [33] pret and execute complex multi-step instructions from users.”
AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
Wang, H., Poskitt, C. M., & Sun, J. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764546 · Cited by 14
“While LLM agents hold significant potential, their autonomous decision-making can sometimes diverge from user expectations, leading to misaligned execution behaviors and raising concerns about their safety and trustworthiness [8, 15, 38, 45, 47, 50].”
Belcastro, L., Carlucci, C., Cosentino, C., Lió, P., & Marozzo, F. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.108160 · Cited by 22
Beyond automation: Unveiling the potential of agentic intelligence
Tirulo, A., Yadav, M., Lolamo, M., Chauhan, S., Siano, P., & Shafie‐khah, M. (2025) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2025.116218 · Cited by 28
A comprehensive introspection on AI risks: taxonomy, challenges, and future directions
Ray, P. P. (2026) · Iran Journal of Computer Science · doi:10.1007/s42044-025-00370-5 · Cited by 16
“When attackers deliberately introduce flaws or exploit latent vulnerabilities in these resources, they can distort the information flow, causing agents to misinterpret data, undertake unintended operations, or pursue adversarial objectives [72, 73].”
A Study on Prompt Injection Attack Against LLM-Integrated Mobile Robotic Systems
Zhang, W., Kong, X., Dewitt, C., Bräunl, T., & Hong, J. B. (2024) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew63542.2024.00103 · Cited by 48
“According to Deng et al. [11], prompt injection attacks exploit various vulnerabilities in LLM systems, threatening their security.”
BackgroundA Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11
“Such incidents highlight how tool chains introduce new classes of security risks: (cid:2) Irreversible execution: Unlike language outputs, tool actions often cannot be undone, turning inference errors into infrastructure failures [33], [131].”
Gasmi, T., Guesmi, R., Bennaceur, J., & Belhadj, I. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123231 · Cited by 10
Tian, J. (2025) · Education Sciences · doi:10.3390/educsci15111540 · Cited by 12
Ke, Y., Jin, L., Elangovan, K., Ong, B. S., Oh, C. Y., Sim, J. X. L., Loh, K., Soh, C. R., Cheng, J., Lee, A. K. Y., et al. (2025) · Anaesthesia · doi:10.1111/anae.16755 · Cited by 19
Saleem, I. (2025) · Higher Education Skills and Work-based Learning · doi:10.1108/heswbl-08-2025-0330 · Cited by 12
Li, L., Zhang, Y., Makanda, I. L. D., & Jiang, P. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103214 · Cited by 11
TrapNet: Model Inversion Defense via Trapdoor
Ma, W.-L., Wang, D.-R., Song, Y., Xue, M.-H., Wen, S., Li, Z.-D., & Xiang, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3560557 · Cited by 11
“I N TODAY’s era of deep neural networks (DNNs), where these models play pivotal roles in privacy-sensitive applications like facial recognition and medical diagnoses [9], [18], [38], [45], the question of safeguarding privacy has gained paramount importance [14], [31], [59].”
Self-citationExternalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1
AI-agent communication network for 6G: vision, architecture, and key technologies
Duan, X., Huang, Z., Liang, S., Zheng, S., Lu, L., & Sun, T. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2500582 · Cited by 8
Adversarial Machine Learning: A 20-Year Survey of Attacks, Defenses, and Standards
Tekeste, B., Al-Hussaeni, K., Fung, B. C. M., Alawadhi, I. M., & Fachkha, C. (2026) · IEEE Access · doi:10.1109/access.2026.3690620 · Cited by 9
“Deng et al. [68] studied the security of autonomous and agentic AI systems, which operate in highly dynamic and complex environments.”
Reliability and security: from swarm robots to AI agents
Yan, Y., Xie, Y., Tang, J., Li, Y., & Jin, Y. (2025) · Journal of Reliability Science and Engineering · doi:10.1088/3050-2454/adea7a · Cited by 6
Towards Verifiably Safe Tool Use for LLM Agents
Doshi, A. M., Hong, Y., Xu, C., Kang, E., Kapravelos, A., & Kästner, C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786839 · Cited by 66
“Safety and security are widely recognized as key obstacles to deploying agents beyond exploratory use [9].”
CyberLLaMA: A fine-tuned large language model for cybersecurity named entity recognition
Zhang, H., Wu, T., Zhu, T., Wen, S., & Xiang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114183 · Cited by 7
Self-citationCollusion-Resistant Multi-Replica Data Auditing With Optimized Metadata Generation
Li, Y., Shen, J., Ji, S., Wen, S., Zhu, T., & Xiang, Y. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3606487 · Cited by 5
Self-citationSoK: Security and Privacy of AI Agents for Blockchain
Romandini, N., Mazzocca, C., Otsuki, K., & Montanari, R. (2025) · International Conference on Blockchain Computing and Applications (BCCA) · doi:10.1109/bcca66705.2025.11229689 · Cited by 16
“While other surveys highlight security and privacy issues of AI agents [14], [15], they do not explicitly refer to agents employed for interacting with blockchain-based environments.”
6G AI Security: From Fundamentals to Offensive and Defensive Landscape in 6G
Braeken, A., Deac, D., Nguyen, T. T., Gür, G., Pham, Q., Yapa, C., Naranjo, P. G. V., Mora, H. C., Moremada, C., & Liyanage, M. (2026) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2026.3659793 · Cited by 10
“With the emergence of LLM, [30] highlights new threats to AI agents, existing countermeasures, and their limitations.”
Detection and analysis of prompt injection in indian multilingual large language models
Srinivasan, J., Regi, S. A., Anbarasan, A. K., Suresh, A., Vetriselvi, T., & Venu, S. (2026) · Scientific Reports · doi:10.1038/s41598-026-43883-0 · Cited by 4
“…from unpredictable inputs from the users, complex internal reasoning chains, environmental variability and interactions with untrusted entities, wherein these prompt injection attacks and jailbreak attacks can compromise perception and decision making pipelines as discussed by Deng et al. 19 .”
Alqahtani, H., & Ahuja, P. (2026) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2026.111184 · Cited by 12
TrustDedup: Secure data deduplication for IoT based on end–edge–cloud collaboration
Yao, X., Li, C., Guo, J., Huang, K., Yao, T., & Zhao, M. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103541 · Cited by 3
Tan, T., & Chen, X. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10770-7 · Cited by 6
“The increasing use of LLMs in security-critical applications has raised awareness of their susceptibility to both misuse and external threats (Deng et al. 2025a, b, c, d).”
Zhang, H., Li, Y., Zhu, T., Han, C., Wen, S., & Xiang, Y. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104255 · Cited by 3
Self-citationWamba-Taguimdje, S.-L., & Kamdjoug, J. R. K. (2026) · Journal of the Operational Research Society · doi:10.1080/01605682.2025.2612146 · Cited by 7
“GenAI agents are intelligent systems built using advanced tools that integrate Machine Learning (ML), Deep Learning (DL), and Natural Language Processing (NLP), GenAI, and autonomous decision-making to interact dynamically and adaptively with environments (Deng et al., 2025; Vuong & Ho, 2024).”
InfluentialSecuring the low-altitude economy: a survey
Yan, M., Dong, R., Han, Q., Deng, Z., Ma, W., Zhu, X., Zhou, W., Wen, S., & Xiang, Y. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4811-2 · Cited by 5
“…5G trial linked an autonomous vehicle over 1000 km away with end-to-end latency under 0.2 s. (2) Artificial intelligence (AI): Onboard AI algorithms handle tasks like navigation [7], obstacle avoidance [8], and target recognition [9], enabling high levels of autonomy [10, 11] in UAV operations.”
InfluentialSelf-citationUnderstanding AI Agents—A Data-Driven Literature Review
Stübinger, J., & Metz, F. (2026) · Mathematics · doi:10.3390/math14091478 · Cited by 7
A real world attack model combining LED modulation and attention-superpixel guidance
Jiang, Y. H., Luo, Y., Fang, J., Jiang, C., Shen, Y., Liao, Y., Sun, Y., Yiu, S., Liu, C., & Jiang, Z. L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128589 · Cited by 3
Chaotic variational auto encoder-based adversarial machine learning
Reddy, D. P. V. S., Vivek, Y., Pranay, G., & Ravi, V. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110646 · Cited by 9
Wang, B., Wang, J., & Li, M. (2025) · Sensors · doi:10.3390/s25134082 · Cited by 2
“Malfunctioning or malicious behaviors may result in legal violations or systemic financial instability [16].”
Untargeted Poisoning Membership Inference With Sample Selection and Enhancement
Li, J., Yang, X., Ma, W., Wang, B., Wen, S., & Xiang, Y. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3601844 · Cited by 2
“T RAINING machine learning (ML) models requires extensive collections of high-quality samples, often sourced from the open world, which can inadvertently include malicious samples [1], [2], [3].”
Self-citationFrom Prompt to Action: A Comprehensive Review of LLM Autonomous Agents
Rafique, Z., Wasim, M., Hussain, M., Hussain, M., & Memon, M. I. (2025) · IEEE International Conference on Wireless for Space and Extreme Environments (WiSEE) · doi:10.1109/wisee57913.2025.11229863 · Cited by 2
Yaacoub, J.-P. A., Noura, H., Salman, O., & Pujolle, G. (2025) · Annals of Telecommunications · doi:10.1007/s12243-025-01134-9 · Cited by 4
“Security challenges include Malicious prompts that bypass filters, produce offensive content, or reveal confidential system instructions represent a security risk to LLMs. Additionally, there is a chance that private or proprietary data may be leaked [219].”
Securing AI and Agentic Systems in Medical Devices: Methods, Risks, and Defenses
Malpure, S., Kanungo, R., Loomis, B. R., & Reza, S. M. S. (2025) · International Conference on Cyber Physical Systems, Power Electronics and Electric Vehicles (ICPEEV) · doi:10.1109/icpeev67897.2025.11291555 · Cited by 2
Yu, P., Liu, B., Tang, S., Li, D., & Lin, W. (2026) · Future Generation Computer Systems · doi:10.1016/j.future.2026.108415 · Cited by 2
Too human to model: the uncanny valley of large language models in simulating human systems
Zeng, Y., Brown, C., & Rounsevell, M. (2026) · npj Complexity · doi:10.1038/s44260-026-00075-1 · Cited by 12
Yang, L., Li, S., & Zhu, C. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104393 · Cited by 2
Xu, C., Zhao, G., Tang, J., Tong, Y., & Yang, Y. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116355 · Cited by 2
OpenClaw in the Wild: Security Analysis of Autonomous Agents
Ma, W., Han, Q., Zhu, X., Zhou, W., Xiong, J., Ren, Z., Wen, S., & Xiang, Y. (2026) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2026.126209 · Cited by 11
Self-citationTraceCaps: Inline Provenance and Risk Enforcement for Agentic Software Engineering
Catarino, A., Mamede, C., Melo, R., & Abreu, R. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786832 · Cited by 3
Gali, M., Kassenkhan, A. M., Chinibayev, Y., Abshukirova, A. M., & Serbin, V. V. (2026) · Technologies · doi:10.3390/technologies14070451 · Cited by 3
Tschurtschenthaler, K., & Schulte, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93721-7_15 · Cited by 2
Development of AI Agent Based on Large Language Model Platforms
Chen, J., & Peng, Y. (2025) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd64986.2025.11082094 · Cited by 2
“However, the development of agents based on LLM still faces many challenges[16].”
Asgari, A., Panichella, A., Derakhshanfar, P., & Olsthoorn, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6234076 · Cited by 2
Louck, Y., Stulman, A., & Dvir, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3821216 · Cited by 2
“Drawing on recent research [7], [8], [13], we identified key vulnerabilities and proposed protocol-level enhancements including short-lived tokens, granular scopes, strong customer authentication, explicit consent orchestration, direct data flow control, and compliance-aware execution states.”
InfluentialA Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis
Spieser, J., Balapour, A., Meller, J., Patra, K. C., & Shamsaei, B. (2026) · Methods and Protocols · doi:10.3390/mps9020033 · Cited by 5
“Already, expanded attack surfaces and emergent risky behaviors have been observed in autonomous agent experiments [49–51].”
Leveraging Human-Centered AI Framework to Mitigate Security, Privacy, and Ethical Risks in AI Agents
Batra, L., Batra, S., & Kathuria, V. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-94159-7_2 · Cited by 1
Adaptive Temporal Action Localization in Video
Xu, Z., Lu, Z., Ding, Y. S., Tian, L., & Liu, S. (2025) · Electronics · doi:10.3390/electronics14132645 · Cited by 5
Defining dangerous AI: existential risk, power-intelligence, and the limits of AGI
Sass, R. (2025) · AI and Ethics · doi:10.1007/s43681-025-00790-w · Cited by 4
“Perhaps less scrupulous actors—or desperate nations engaged in military conflict—may allow autonomous AI agents to “self-update” or “self-patch” without extensive auditing [57, 58].”
Jiang, Y. H., Wang, Y., Luo, Y., Yang, Z., Liao, Y., Shen, Y., Lin, Y. P., Pan, H., Jiang, Z. L., & Fang, J. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3599975 · Cited by 1
“However, due to the noninterpretable nature of deep neural networks, lane detection models still face security threats posed by adversarial examples [14], [15], [16].”
ABACPR: Attribute-Based Access Control Supporting Policy Reconstruction for Intent-Based Networking
Zhang, T., Yang, H., Shen, J., Vijayakumar, P., Arya, V., & Gupta, B. B. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3621538 · Cited by 1
“Therefore, it remains a challenge to achieve highly dynamic and secure data sharing in the IoT communication [5], [6], [7].”
Failure Is Not an Option: It’s Reality Calling
Wallace, K. (2025) · Computer · doi:10.1109/mc.2025.3605057 · Cited by 2
“At scale and in what will likely be an unforgiving environment, 26 the challenges presented by such autonomous systems with respect to safety are significant, particularly if large language models are the go-to solution to provide the required intelligence.”
PmiaNLL: Defending against poisoning membership inference attacks with noisy label learning
Li, J., Yang, X., Ma, W., Wang, B., Zhu, T., Wen, S., & Xiang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114706 · Cited by 1
Self-citationPolyp image segmentation based on parallel dilated convolution and dual attention mechanisms
Chen, S., Chen, K., Wang, C., Zhou, Z., Wen, S., Zhu, T., & Wu, M. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108282 · Cited by 1
Self-citationTransLock: Securing LLM deployment for software applications via self-locking watermarks
Ye, P.-G., Zhang, Z., Mo, K., & Peng, Y. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10750-x · Cited by 1
“The transformative potential of Large Language Models (LLMs) (Deng et al. 2025; Zhu et al. 2025; Zhou et al. 2025; Deng et al. 2025) has sparked adoption across various industry verticals, from finance (Yu et al. 2024) and healthcare (Goyal et al. 2024) to legal (Cheong et al. 2024) and…”
Enhancing DNN Adversarial Robustness via Dual Stochasticity and Geometric Normalization
Wu, X., & Han, G. (2025) · Sensors · doi:10.3390/s25237398 · Cited by 1
“The comprehensive survey “Threatened Artificial Intelligence Agents: Key Security Challenges and Future Directions” [7] further highlights that adversarial attacks have become one of the main security risks faced by artificial intelligence systems, exacerbating concerns about their credibility and…”
AI Agents: A Comprehensive Review of Evolution, Architectures, Applications, and Future Directions
Akuma, C. F., Hewage, P., Garg, A., & Amaechi, C. V. (2025) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito66076.2025.11241897 · Cited by 1
Kim, D. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108022 · Cited by 4
Artificial Intelligence Agents for Sustainable Production Based on Digital Model-Predictive Control
Bakhtadze, N., Dozortsev, V. M., Vlasov, A., Koroleva, M. S., & Anikin, M. (2026) · Sustainability · doi:10.3390/su18020759 · Cited by 1
uulu, C. U., Kulyabin, M., Fuhrmann, I., Joosten, J., Pacheco, N. M. M., Petridis, F., Johnson, R., Bosch, J., & Olsson, H. H. (2026) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3793653.3793776 · Cited by 1
“LLM based scientific visualization We define an AI agent following [7] as a system exhibiting four key properties: autonomy (operates independently after prompting), reactivity (responds to user requests), proactivity (applies expert rules), and social ability (natural language interaction).”
A Case Study on Delegating Critical Tasks to Agentic AI and Prototype Access Control Methods
Kim, S., & Kim, H. (2025) · Cyber Awareness and Research Symposium (CARS) · doi:10.1109/cars67163.2025.11337816 · Cited by 1
“1, such capabilities often require users to provide the AI with sensitive information, such as credit card information, address, or social security numbers, creating serious security and privacy risks [5].”
Contextualizing AI Agent Evaluation: Proposed Framework for Japanese Businesses
Toyoda, R., Kiyomoto, H., Komayama, S., Shigetani, H., & Fukui, M. (2025) · International Conference on Artificial Intelligence for Sustainable Innovation (AI-SI) · doi:10.1109/ai-si66213.2025.11341698 · Cited by 1
“AI agents, especially large language models (LLMs)-based agents are increasingly being implemented across several industries [1]– [3].”
Securing LLM-Generated Embedded Firmware Through AI Agent-Driven Validation and Patching
Abtahi, S. M., & Azim, A. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00044 · Cited by 1
“AI agents offer task-specific capabilities that complement LLMs by collaboratively addressing security and performance in firmware pipelines [7], [8].”
Secure Agent-Based Architectures for Decentralized AI Identity Management: The DAIS Framework
Ragupathy, V. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395719 · Cited by 1
“These standards are for human-centric identification models and do not address autonomous AI agent operational realities like extremely high verification frequency, dynamic authorisation needs, situational context change, and Byzantine adversarial behaviour resistance [11].”
Mikhaylov, A. Y., Ikramov, M. A., Nabieva, N. M., Sokolov, B., Zhang, W., Nazarov, V., Ergasheva, M., Turaev, S., Meilyeva, D., Dinets, D., et al. (2026) · Sound&Vibration · doi:10.59400/sv3941 · Cited by 1
“Frontline service challenges, such as user trust and the handling of complex queries, are also a key area of concern and opportunity [7–9].”
Breaking Barriers: Multi-Agent Prompt Fusion for Automated LLM Jailbreaks
Hu, J., Zhang, J., & Li, Z. (2026) · Cognitive Computation · doi:10.1007/s12559-026-10552-3 · Cited by 2
Characterizing Unintended Consequences of GUI Agents For Web Browsing
Zhang, S., Chen, J., Gao, Z., Gao, J., Yi, X., & Li, H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790696 · Cited by 18
AIBlockAgent: Design and Development of an AI Agent to Support Block-Based Programming Learning
Choa, H., & Koo, D. (2026) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2026.29.3.011 · Cited by 1
Ndayishimiye, R., Gentil, B., & Libertas, G. A. (2026) · Intelligent Control and Automation · doi:10.4236/ica.2026.172003 · Cited by 1
Chen, S., Xiong, L., Chow, M., & Tang, Y. (2026) · Cyber-Physical Energy Systems · doi:10.1016/j.cpes.2026.04.003 · Cited by 1
Evolution but Still Exposed: An Empirical Study of Residual APIs in the ChatGPT Plugin Ecosystem
Ren, R., Yan, C., Dong, N., & Bai, G. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805537 · Cited by 2
“More broadly, plugins and protocol adapters in the LLM agent ecosystem often depend on ad hoc authentication mechanisms and inconsistent validation practices, indicating that insufficient security governance of third-party backends is a cross-ecosystem problem [12, 32, 33].”
Zhuhadar, L. (2026) · Future Internet · doi:10.3390/fi18070376 · Cited by 3
Abdullah, M., Hongying, Z., Javed, A., Wahid, J. A., Ahmed, K., Eshkiki, H., & Caraffini, F. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02791-5 · Cited by 1
AgentDID: Trustless Identity Authentication for AI Agents
Xu, M., Liu, X., Guo, Y., Liu, C., Zhang, Y., & Cheng, X. (2026) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/2575-8411.2026.00116 · Cited by 1
“However, the rapid expansion of the AI agent ecosystem has revealed critical security gaps [2].”
Self-citation
Metadata
- Authors
- Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y.
- Affiliations
- Swinburne University of Technology (AU)
- Tsinghua University (CN)
- Ant Group (China) (CN)
- Venue
- ACM Computing Surveys
- Published
- 2025-02-21
- Volume
- 57
- Issue
- 7
- Pages
- 1-36
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4407236106
- Cited by
- 210
- References
- 92
- Citations by year
- 2026: 127 · 2025: 82
- FWCI
- 83.83
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Computer Networks and Communications › Access Control and Trust
- Keywords
- AI agents, AI agent security, security threats, autonomous software agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.