The DOI matches its Crossref record; 264 of the 288 references it cites were found in a registry or an index.
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Open accessAbstract
Agentic AI systems powered by large language models (LLMs) and endowed with planning, tool use, memory, and autonomy, are emerging as powerful, flexible platforms for automation. Their ability to autonomously execute tasks across web, software, and physical environments creates new and amplified security risks, distinct from both traditional AI safety and conventional software security. This survey outlines a taxonomy of threats specific to agentic AI, reviews recent benchmarks and evaluation methodologies, and discusses defense strategies from both technical and governance perspectives.We synthesize current research and highlight open challenges, aiming to support the development of secure-by-design agent systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second is the development of agentic AI: software systems that wrap LLM reasoning in perception, planning, memory, tool-use, and multi-agent communication loops, enabling them to autonomously execute multi-step security tasks that previously required human analysts”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They introduce new attack surfaces: prompt injection via untrusted observations [88], tool hijacking, and emergent collusion in multiagent settings [22,26].”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“across the full 153-paper corpus, only five studies (≈3%)- [22,71,125] for error propagation and [26,87] for tool-use amplification-report observing a compound failure mode at all”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hao, X.-B., & Zhang, G.-G. (2017) · Encycl. Semantic Comput. Robotic Intell. · doi:10.1142/s2425038416300184 · Cited by 86,401 · cited 2×
“…can have across diverse domains, these systems are being increasingly employed in several real-world scenarios and applications: • Automation of complex workflows: Agentic AI systems can autonomously manage supply chain processes and adapt procurement decisions in response to dynamic data…”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“For instance, Devin , an AI software engineer developed by Cognition Labs, has demonstrated the ability to plan, code, debug, and deploy software with minimal oversight.”
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“Analogous to defending against traditional ML/AI adversarial examples in computer vision, completely preventing prompt injection is an open problem [186].”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Modern pretraining and supervised fine-tuning paradigms on dialogue-style data, which trains the model to learn short-term instruction-response behavior (while deprioritizing long-term embodied sequential state tracking), are likely resulting in these shortcomings [165, 162].”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1×
“For instance, Devin , an AI software engineer developed by Cognition Labs, has demonstrated the ability to plan, code, debug, and deploy software with minimal oversight.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“In the subsequent section, we discuss these motivations in more detail, and delineate our contributions towards augmenting the safety and security of agentic AI frameworks.”
Robust physical-world attacks on deep learning visual classification
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00175 · Cited by 2,173
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
Athalye, A., Carlini, N., & Wagner, D. (2018) · arXiv · doi:10.48550/arxiv.1802.00420 · Cited by 2,172 · cited 1×
“More generally, in adversarial attack research in ML/AI, many defenses initially reported as effective are commonly undermined soon after their release, primarily because they lack thorough evaluation against adaptive attack strategies [182, 251, 252].”
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15594 · Cited by 1,781 · cited 1×
“…maintain safe behavior across multi-step subgoals, interruptions, and/or distribution shifts; and (ii) latent misbehavior detection : recent work [246] has shown that deceptive policies can persist through standard safety training and only activate under triggers, undermining short-episode audits.”
Secure, privacy-preserving and federated machine learning in medical imaging
Kaissis, G. A., Makowski, M. R., Rückert, D., & Braren, R. F. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0186-1 · Cited by 1,676
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“In related efforts, Chen et al.[207] and Siddiq et al. [208] employ containerized environments (e.g., gVisor or Docker) to execute LLM-generated code against unit tests, thereby confining execution vulnerabilities without exposing the host system.”
A Survey of Defense Mechanisms Against Distributed Denial of Service (DDoS) Flooding Attacks
Zargar, S. T., Joshi, J., & Tipper, D. (2013) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2013.031413.00127 · Cited by 1,396 · cited 1×
“We also delineate potential shortcomings of these approaches.”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257 · cited 1×
“• Personalized support: When equipped with memory and adaptive reasoning, agentic systems can function as persistent digital concierges that track user preferences over time.”
Data Poisoning Attacks Against Federated Learning SystemsPreprint
Tolpegin, V., Truex, S., Gursoy, M. E., & Liu, L. (2020) · European Symposium on Research in Computer Security · doi:10.48550/arxiv.2007.08432 · Cited by 1,031 · cited 1×
“Other work on retrieval corruption and federated training manipulation, studies scenarios where adversaries distort contextual inputs in retrieval-augmented generation pipelines [117] or inject malicious updates in federated learning settings to corrupt distributed training data [118].”
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
Carlini, N., & Wagner, D. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140444 · Cited by 966 · cited 1×
“More generally, in adversarial attack research in ML/AI, many defenses initially reported as effective are commonly undermined soon after their release, primarily because they lack thorough evaluation against adaptive attack strategies [182, 251, 252].”
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838 · cited 1×
“Prompts are commands that specify how an AI agent should behave [56].”
Agentic AI: Autonomous intelligence for complex goals–a comprehensive survey
Acharya, D. B., Kuppan, K., & Divya, B. (2025) · IEEE Access · doi:10.1109/access.2025.3532853 · Cited by 717 · cited 1×
“Existing surveys on autonomous AI agents [49–53] mostly detail their capabilities and propose evaluation benchmarks, while others explore topics such as trust and risk management [54] without a singular focus on security, unlike our work.”
The epistemology of a rule-based expert system —a framework for explanation
Clancey, W. J. (1983) · Artificial Intelligence · doi:10.1016/0004-3702(83)90008-5 · Cited by 712 · cited 1×
“…the potential for positive impact agentic AI can have across diverse domains, these systems are being increasingly employed in several real-world scenarios and applications: • Automation of complex workflows: Agentic AI systems can autonomously manage supply chain processes and adapt procurement…”
Fooling Automated Surveillance Cameras: Adversarial Patches to Attack Person Detection
Thys, S., Ranst, W. V., & Goedeme, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00012 · Cited by 694
Unmanned Aircraft Capture and Control Via GPS Spoofing
Kerns, A. J., Shepard, D. P., Bhatti, J. A., & Humphreys, T. E. (2014) · Journal of Field Robotics · doi:10.1002/rob.21513 · Cited by 688
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686 · cited 1×
“Similarly, in [180], the authors introduce SecAlign , which leverages alignment training via direct preference optimization (DPO) [181] to align agents toward preferring benign instructions over adversarial ones.”
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667 · cited 1×
“AI Prompt injection (PI) remains the most widely discussed attack in the literature, where malicious instructions cause the model to deviate from intended behavior [57, 58].”
Otter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 655 · cited 1×
“• Personalized support: When equipped with memory and adaptive reasoning, agentic systems can function as persistent digital concierges that track user preferences over time.”
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630 · cited 2×
“Unlike static attacks (e.g. traditional malware), these attacks exploit the very framework of multi-agent systems [133, 134].”
Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models
Yi, J., Xie, Y., Zhu, B., Kıcıman, E., Sun, G., Xie, X., & Wu, F. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709179 · Cited by 627
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024) · arXiv · doi:10.48550/arxiv.2406.13352 · Cited by 616 · cited 2×
“A simple approach is requiring the LLM to commit to a predefined set of tools for a task, with the system controller disabling access to others [190, 59].”
(1995) · Manufacturing Engineer · doi:10.1049/me:19950308 · Cited by 601 · cited 1×
“For example, U.S. manufacturers such as Toro have deployed AI systems that analyze tariff policies, commodity pricing, and economic signals to recommend procurement actions, leaving operators with a simple acceptance decision [21].”
On the requirements for successful GPS spoofing attacks
Tippenhauer, N. O., Pöpper, C., Rasmussen, K. B., & Capkun, S. (2011) · Proceedings of the 18th ACM conference on Computer and communications security · doi:10.1145/2046707.2046719 · Cited by 506
A Dynamic LLM-Powered Agent Network for Task-Oriented Agent CollaborationPreprint
Liu, Z., Zhang, Y., Li, P., Liu, Y., & Yang, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.02170 · Cited by 465 · cited 1×
“Prompt injection remains one of the most persistent attack vectors against LLM agents, as adversarial inputs can override intended behavior and subvert downstream actions [63].”
Empowering Biomedical Discovery with AI AgentsPreprint
Gao, S.-H., Fang, A., Huang, Y.-P., Giunchiglia, V., Noori, A., Schwarz, J., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.48550/arxiv.2404.02831 · Cited by 428 · cited 1×
“This line of research highlights the potential of agentic AI in accelerating scientific progress and expanding the knowledge frontier [24, 25].”
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14499 · Cited by 370
SecAlign: Defending Against Prompt Injection with Preference Optimization
Chen, S., Zharmagambetov, A. S., Mahloujifar, S., Chaudhuri, K., Wagner, D., & Guo, C. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744836 · Cited by 366 · cited 2×
“Similarly, in [180], the authors introduce SecAlign , which leverages alignment training via direct preference optimization (DPO) [181] to align agents toward preferring benign instructions over adversarial ones.”
AutoAgents: A Framework for Automatic Agent Generation
Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B. F., Fu, J., & Shi, Y.-M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2309.17288 · Cited by 326 · cited 1×
“In the subsequent section, we discuss these motivations in more detail, and delineate our contributions towards augmenting the safety and security of agentic AI frameworks.”
Are Large Pre-Trained Language Models Leaking Your Personal Information?Preprint
Huang, J., Shao, H., & Chang, K. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.12628 · Cited by 315 · cited 1×
“Complementarily, known-answer detection uses cryptographic tokens embedded in user commands; if the LLM fails to return the token, this signals a system-wide prompt injection compromise [185].”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293 · cited 1×
“Similarly, in [180], the authors introduce SecAlign , which leverages alignment training via direct preference optimization (DPO) [181] to align agents toward preferring benign instructions over adversarial ones.”
Automatic and Universal Prompt Injection Attacks against Large Language ModelsPreprint
Liu, X., Yu, Z., Zhang, Y., Zhang, N., & Xiao, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.04957 · Cited by 285 · cited 1×
“Originally proposed for jailbreaking, Greedy Coordinate Gradient (GCG) [64] has been adapted to IPI by generating affirmative prefixes containing adversarial strings that induce malicious outputs from the agent [65].”
Transferable Adversarial Attacks for Image and Video Object DetectionPreprint
Wei, X., Liang, S.-Y., Cao, X.-C., & Zhu, J. (2018) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.1811.12641 · Cited by 280 · cited 1×
“…, where a single malicious prompt triggers a chain of compromised behaviors across future interactions [61, 38, 95], and autonomous propagation , which includes multi-agent infections and AI worms that spread malicious prompts across agents or system boundaries without user intervention [96, 38].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276
Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection
Yan, J., Yadav, V., Li, S., Chen, L., Tang, Z. S., Wang, H., Srinivasan, V., Ren, X., & Jin, H. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.337 · Cited by 275 · cited 1×
“Malicious or unverified agents can be introduced during runtime as part of task-specific teams, as evidenced by backdoor attacks [129–132].”
Artificial intelligence risk management framework :
National Institute of Standards and Technology (US) (2024) · NIST Trustworthy and Responsible AI · doi:10.6028/nist.ai.600-1 · Cited by 274
PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in ActionPreprint
Shao, Y., Li, T.-S., Shi, W.-Y., Liu, Y., & Yang, D. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2409.00138 · Cited by 261
Illusion and Dazzle: Adversarial Optical Channel Exploits Against Lidars for Automotive Applications
Shin, H., Kim, D., Kwon, Y., & Kim, Y. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66787-4_22 · Cited by 247
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Chen, Z., Xiang, Z., Xiao, C., Song, D., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.12784 · Cited by 246
Poisoning Retrieval Corpora by Injecting Adversarial PassagesPreprint
Zhong, Z., Huang, Z., Wettig, A., & Chen, D. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.19156 · Cited by 238 · cited 1×
“Note that IPI attacks take advantage of the agent’s dependence on outside tools and information sources by enclosing harmful tasks in resources that appear to be harmless, such as databases, APIs, or web pages [68–71].”
BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning
Jia, J., Liu, Y., & Gong, N. (2021) · IEEE Symposium on Security and Privacy · doi:10.48550/arxiv.2108.00352 · Cited by 231
AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
Zhu, S., Zhang, R., An, B., Wu, G., Barrow, J., Wang, Z., Huang, F., Nenkova, A., & Sun, T. (2023) · arXiv · doi:10.48550/arxiv.2310.15140 · Cited by 230 · cited 1×
“Lastly, as the aforementioned attacks often generate gibberish strings that can be easily detected via perplexity defenses, AutoDAN [67] enhances the semantic quality of adversarial strings to decrease detectability.”
Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
Lee, D., & Tiwari, M. (2024) · arXiv · doi:10.48550/arxiv.2410.07283 · Cited by 228 · cited 10×
“In comparison to unintentional prompt injection, as purposeful attacks aim to force agents to carry out harmful behaviors rather than only produce dangerous outputs, they necessitate proactive protection strategies such as input sanitization, tool-use monitoring, and anomaly detection [63].”
InfluentialIron: Private Inference on Transformers
Hao, M., Li, H.-W., Chen, H.-X., Xing, P., Xu, G., & Zhang, T.-W. (2022) · Neural Information Processing Systems · doi:10.52202/068431-1143 · Cited by 224 · cited 1×
“Even in privacy-preserving deployments where inputs and outputs remain encrypted [154–156], attackers may exploit the lack of complete plaintext visibility across domains.”
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222 · cited 1×
“• Revolutionizing healthcare: Agentic AI is transforming healthcare by autonomously personalizing treatment, streamlining both clinical and administrative workflows, and enhancing patient support.”
Cascella, M., Semeraro, F., Montomoli, J., Bellini, V., Piazza, O., & Bignami, E. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02045-3 · Cited by 219 · cited 1×
“• Personalized support: When equipped with memory and adaptive reasoning, agentic systems can function as persistent digital concierges that track user preferences over time.”
Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMsPreprint
Bagdasaryan, E., Hsieh, T.-Y., Nassi, B., & Shmatikov, V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10490 · Cited by 200 · cited 4×
“Past work has demonstrated how adversarial perturbations can be used to blend malicious prompt injections in audio content to undertake IPI attacks [81] and jailbreak models [90].”
InfluentialDataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks
Liu, Y., Jia, Y., Jia, J., Song, D., & Gong, N. Z. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00250 · Cited by 199 · cited 1×
“(1) New distributional coverage metrics that score trajectory segments, as opposed to only considering the end-state (i.e. whether a task was completed successfully) and capture the entire distribution of performance (instead of average performance or a single run).”
Universal and Transferable Adversarial Attacks on Aligned Language Models
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197 · cited 1×
“Originally proposed for jailbreaking, Greedy Coordinate Gradient (GCG) [64] has been adapted to IPI by generating affirmative prefixes containing adversarial strings that induce malicious outputs from the agent [65].”
Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM AgentsPreprint
Zhan, Q., Fang, R., Panchal, H., & Kang, D. (2025) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2503.00061 · Cited by 196 · cited 6×
“Prompt injection remains one of the most persistent attack vectors against LLM agents, as adversarial inputs can override intended behavior and subvert downstream actions [63].”
InfluentialAutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08155 · Cited by 193 · cited 1×
“Knowledge and Learning Manipulation Cross-domain multi-agent LLMs allow agents to self-improve via shared learning and distributed fine-tuning [109, 135].”
The BrowserGym Ecosystem for Web Agent Research
Chezelles, T. L. S. D., Gasse, M., Lacoste, A., Caccia, M., Boisvert, L., Thakkar, M., Marty, T., Assouel, R., Shayegan, S. O., Jang, L., et al. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2412.05467 · Cited by 186
Black-box Adversarial Attacks on Video Recognition Models
Jiang, L., Ma, X., Chen, S., Bailey, J., & Jiang, Y. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351088 · Cited by 177 · cited 1×
“…, where a single malicious prompt triggers a chain of compromised behaviors across future interactions [61, 38, 95], and autonomous propagation , which includes multi-agent infections and AI worms that spread malicious prompts across agents or system boundaries without user intervention [96, 38].”
Next-generation agentic AI for transforming healthcare
Karunanayake, N. (2025) · Informatics and Health · doi:10.1016/j.infoh.2025.03.001 · Cited by 165
The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM AgentsPreprint
Jia, F., Wu, T., Qin, X., & Squicciarini, A. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2412.16682 · Cited by 163 · cited 1×
“At the system level, tool filtering [190] restricts agent calls to predefined tool sets, and TaskShield [197] validates tool-use alignment with user intent.”
Harms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149 · cited 1×
“BrowserGym brings many such tasks under one consistent interface and scoring protocol (e.g., MiniWoB++, WorkArena, WebArena, among others), reducing fragmentation and enabling like-for-like comparisons across LLMs and agent designs [223].”
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent TrajectoriesPreprint
Lù, X. H., Kazemnejad, A., Meade, N., Patel, A., Shin, D., Zambrano, A., Stańczak, K., Shaw, P. A., Pal, C., & Reddy, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.08942 · Cited by 148 · cited 1×
“Security evaluation often requires testing unsafe prompts or tool sequences; executing these against live systems is risky and irreproducible [210, 232, 239–241].”
Attention Tracker: Detecting Prompt Injection Attacks in LLMsPreprint
Hung, K.-H., Ko, C.-Y., Rawat, A., Chung, I.-H., Hsu, W. H., & Chen, P.-Y. (2024) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2411.00348 · Cited by 146 · cited 1×
“Recent studies show that adaptive attacks have been largely unexplored and ignored, as most work undertakes static (i.e., non-adaptive) defense evaluations for AI agents [63, 182, 250].”
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · arXiv · doi:10.48550/arxiv.2403.02691 · Cited by 143 · cited 2×
“Prompt injection attacks remains a critical LLM vulnerability, permitting adversaries to manipulate agent behavior through crafted inputs [37–39].”
Attacking Vision-Language Computer Agents via Pop-upsPreprint
Zhang, Y., Yu, T., & Yang, D.-Y. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2411.02391 · Cited by 142 · cited 1×
“The challenge is compounded by the fact that successful IPIs frequently exploit the decoupling between user inputs and subsequent tool calls, a behavioral property that differentiates them from traditional jailbreak prompts.”
AIoT Smart Home via Autonomous LLM Agents
Rivkin, D., Hogan, F. R., Feriani, A., Konar, A., Sigal, A., Liu, X., & Dudek, G. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3471904 · Cited by 142
The rise of agentic AI teammates in medicine
Zou, J., & Topol, E. J. (2025) · The Lancet · doi:10.1016/s0140-6736(25)00202-8 · Cited by 138 · cited 1×
“Healthcare AI agents [29–33] have been developed that can continuously monitor chronic conditions for patients, adapt care strategies in real-time, handle patient interactions/follow-ups, act as scribes for reducing clinician documentation burden, and accelerate drug discovery via autonomous…”
Pedro, R., de Castro, D. A., Carreira, P., & Santos, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01990 · Cited by 137 · cited 2×
“These range from direct prompt injection to code injection under the pretense of legitimate programming activities, enabling the generation of malicious code or SQL queries to carry out disallowed actions (known as prompt-to-SQL or P2SQL attacks) [82, 61].”
OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
Kuntz, T. M., Duzan, A., Zhao, H.-C., Croce, F., Kolter, Z., Flammarion, N., & Andriushchenko, M. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.14866 · Cited by 136 · cited 5×
“Emulation (ToolEmu) and VM-backed sandboxes (OSWorld/OS-Harm) provide safe, deterministic environments for repeatable experiments [210, 232].”
InfluentialMind2Web: Towards a Generalist Agent for the Web
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023) · arXiv · doi:10.48550/arxiv.2306.06070 · Cited by 134 · cited 4×
“Complementary initiatives include the NCCoE Cyber AI Profile [220], which offers implementation guidance, Cybersecurity Framework, for integrating cybersecurity controls into AI systems; the OWASP Agentic AI Threats project [221], which catalogs common vulnerabilities such as insecure orchestration and prompt injectio…”
InfluentialRaza, S., Sapkota, R., Karkee, M., & Emmanouilidis, C. (2025) · AI Open · doi:10.48550/arxiv.2506.04133 · Cited by 134 · cited 1×
“Existing surveys on autonomous AI agents [49–53] mostly detail their capabilities and propose evaluation benchmarks, while others explore topics such as trust and risk management [54] without a singular focus on security, unlike our work.”
(2023) · doi:10.52202/075280-1220 · Cited by 134
Current state of LLM Risks and AI GuardrailsPreprint
Ayyamperumal, S. G., & Ge, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.12934 · Cited by 126 · cited 1×
“In practice, industry guidance has begun to outline deployment patterns for such guardrails, emphasizing layered and modular approaches to real-world alignment [198, 199].”
Memory Injection Attacks on LLM Agents via Query-Only Interaction
Dong, S., Xu, S., He, P., Li, Y., Tang, J., Liu, T., Liu, H., & Xiang, Z. (2025) · arXiv · doi:10.48550/arxiv.2503.03704 · Cited by 124
SafeArena: Evaluating the Safety of Autonomous Web AgentsPreprint
Tur, A. D., Meade, N., Lù, X. H., Zambrano, A., Patel, A., Durmus, E., Gella, S., Stańczak, K., & Reddy, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.04957 · Cited by 124 · cited 1×
“…maintain safe behavior across multi-step subgoals, interruptions, and/or distribution shifts; and (ii) latent misbehavior detection : recent work [246] has shown that deceptive policies can persist through standard safety training and only activate under triggers, undermining short-episode audits.”
SALLM: Security Assessment of Generated Code
Siddiq, M. L., Santos, J. C. S., Devareddy, S., & Muller, A. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694934 · Cited by 124 · cited 1×
“For instance, the NIST AI Risk Management Framework (AI RMF) Generative AI Profile [55], developed under Executive Order 14110 , provides a cross-sectoral reference for managing risks specific to generative and agentic AI.”
Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents
Zhan, Q., Fang, R., Panchal, H. S., & Kang, D. (2025) · Findings of the Association for Computational Linguistics: NAACL 2025 · doi:10.18653/v1/2025.findings-naacl.395 · Cited by 124
Hidden Backdoors in Human-Centric Language Models
Li, S., Liu, H., Dong, T., Zhao, B. Z. H., Xue, M., Zhu, H., & Lu, J. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484576 · Cited by 119 · cited 1×
“We also delineate potential shortcomings of these approaches.”
Get My Drift? Catching LLM Task Drift with Activation Deltas
Abdelnabi, S., Fay, A., Cherubin, G., Salem, A., Fritz, M., & Paverd, A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00011 · Cited by 119 · cited 3×
“In particular, this is crucial for security evaluations: a system that is safe in expectation but occasionally executes a destructive action is unacceptable for any mission-critical scenarios.”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., Dobriban, E., Flammarion, N., Pappas, G. J., Tramer, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.01318 · Cited by 116
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., Dobriban, E., Flammarion, N., Pappas, G., Tramèr, F., et al. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1745 · Cited by 116
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Furuta, H., Lee, K.-H., Nachum, O., Matsuo, Y., Faust, A., Gu, S. S., & Gur, I. (2023) · arXiv · doi:10.48550/arxiv.2305.11854 · Cited by 115 · cited 1×
“The τ - BENCH pass ∧ k metric operationalizes reliability under repeated trials with minor stochasticity [227].”
Ignore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110 · cited 1×
“AI Prompt injection (PI) remains the most widely discussed attack in the literature, where malicious instructions cause the model to deviate from intended behavior [57, 58].”
AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web AgentsPreprint
Yang, K., Liu, Y., Chaudhary, S., Fakoor, R., Chaudhari, P., Karypis, G., & Rangwala, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.13825 · Cited by 110 · cited 3×
“These risks are not rooted in the agent’s internal reasoning or learning capabilities, but rather in the mismatch, fragility, or variability of the interfaces and environments through which the agent perceives and acts [162–164].”
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 108 · cited 1×
“We will now discuss some sub-categories of system-focused defenses.”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022) · arXiv · doi:10.48550/arxiv.2207.01206 · Cited by 107
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1508 · Cited by 107
GTA: A Benchmark for General Tool AgentsPreprint
Wang, J.-Z., Ma, Z.-R., Li, Y., Zhang, S.-Y., Chen, C.-L., Chen, K., & Le, X.-Y. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2407.08713 · Cited by 106
Secret Collusion among AI Agents: Multi-Agent Deception via Steganography
Motwani, S., Baranchuk, M., Strohmeier, M., Bolina, V., Torr, P., Hammond, L., & Witt, C. D. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2336 · Cited by 106 · cited 1×
“By spoofing an agent’s identity or posing as a trusted collaborator, an attacker gains entry into workflows that would otherwise be restricted [124, 125].”
Imprompter: Tricking LLM Agents into Improper Tool UsePreprint
Fu, X., Li, S., Wang, Z., Liu, Y., Gupta, R. K., Berg-Kirkpatrick, T., & Fernandes, E. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.14923 · Cited by 102 · cited 1×
“Other agent-specific attacks include memory poisoning , that enables stealthy manipulation over time, as agents retain and act on corrupted context [42–44] and tool misuse , such as abusing calendar or API integrations, which can trigger unintended or malicious actions [45, 46].”
Adversarial objects against lidar-based autonomous driving systems
Cao, Y., Xiao, C., Yang, D., Fang, J., Yang, R., Liu, M., & Li, B. (2019) · arXiv · doi:10.48550/arxiv.1907.05418 · Cited by 102
An Early Categorization of Prompt Injection Attacks on Large Language ModelsPreprint
Rossi, S., Michel, A. M., Mukkamala, R. R., & Thatcher, J. B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.00898 · Cited by 102 · cited 1×
“For instance, in [100], the authors design LLM agents capable of performing intricate multi-step website exploits via strategic combinations of tool calls and dynamic planning.”
LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT PluginsPreprint
Iqbal, U., Kohno, T., & Roesner, F. (2023) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.48550/arxiv.2309.10254 · Cited by 102
Adaptive Reward-Poisoning Attacks against Reinforcement LearningPreprint
Zhang, X., Ma, Y., Singla, A. K., & Zhu, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.12613 · Cited by 100 · cited 1×
“There are two main approaches for doing so: Prompt engineering and runtime output behavior The authors in [176] propose an instruction hierarchy, which establishes priority levels for different instruction sources so that user-provided instructions are always prioritized over potentially malicious…”
Large Language Models: Their Success and Impact
Makridakis, S., Petropoulos, F., & Kang, Y. (2023) · Forecasting · doi:10.3390/forecast5030030 · Cited by 99
Coordinated AI agents for advancing healthcare
Moritz, M., Topol, E. J., & Rajpurkar, P. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01363-2 · Cited by 95
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Berant, J., Cohan, J., Hu, H., Joshi, M., Khandelwal, U., Lee, K., Pasupat, P., Shaw, P., & Toutanova, K. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.00245 · Cited by 89 · cited 1×
“On the long-standing MiniWoB/MiniWoB++ benchmarks, proposed methods have continued to make progress [224–226].”
Heuristic Black-box Adversarial Attacks on Video Recognition Models
Wei, Z.-P., Chen, J.-J., Wei, X., Jiang, L., Chua, T.-S., Zhou, F.-F., & Jiang, Y.-G. (2019) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1911.09449 · Cited by 89
Xu, S. (2024) · AIP Conference Proceedings · doi:10.48550/arxiv.2401.07612 · Cited by 87 · cited 1×
“The open challenge thus is to measure and improve agentic AI safety across very long time/episode horizons .”
Can Indirect Prompt Injection Attacks Be Detected and Removed?Preprint
Chen, Y.-L., Li, H.-R., Sui, Y., He, Y.-F., Liu, Y., Song, Y.-Q., & Hooi, B. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.16580 · Cited by 87 · cited 1×
“Indirect Prompt Injection (IPI) attacks instead cause LLMs to diverge from user-provided instructions by inserting malicious instructions into external data that the model processes [62, 37].”
AgentBench: Evaluating LLMs as Agents
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79
Watermarking Makes Language Models Radioactive
Sander, T., Fernandez, P., Durmus, A., Douze, M., & Furon, T. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2402.14904 · Cited by 79 · cited 1×
“Detection-based defenses Detection-based defenses seek to detect malicious inputs or outputs without modifying the fundamental LLM. Input detection methods often rely on separate filters, such as guardrail models, that screen prompts before they reach the target system [175].”
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74
MCPSecBench: A Systematic Security Benchmark and Playground for Testing Model Context ProtocolsPreprint
Yang, Y., Gao, C., Wu, D., Chen, Y., Li, Y., & Wang, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.13220 · Cited by 73
Prompt-to-SQL Injections in LLM-Integrated Web Applications: Risks and Defenses
Pedro, R., Coimbra, M. E., de Castro, D. A., Carreira, P., & Santos, N. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00007 · Cited by 72 · cited 1×
“We now discuss and classify the works in this domain into two broad categories.”
MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI AgentsPreprint
Zhu, K., Yang, X., Wang, J., Guo, W., & Wang, W. Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.05174 · Cited by 70 · cited 4×
“Other system-focused defenses, such as Melon , employ constrained execution sandboxes and verification loops to limit the impact of potentially malicious instructions on downstream systems, serving as a defense against IPI attacks [72].”
InfluentialLlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and ModelsPreprint
Helff, L., Friedrich, F., Brack, M., Kersting, K., & Schramowski, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.05113 · Cited by 67
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.07972 · Cited by 64 · cited 2×
“OS-Harm builds on OSWorld’s [229] full desktop environment to evaluate agent safety across office applications and file operations, scoring both accuracy and adherence to safety guidelines via LLM-based judges with validated agreement levels to human annotations [232].”
TrustAgent: Towards Safe and Trustworthy LLM-based Agents through Agent Constitution
Hua, W.-Y., Yang, X., Li, Z., Cheng, W., & Zhang, Y.-F. (2024) · arXiv.org · doi:10.48550/arxiv.2402.01586 · Cited by 64 · cited 1×
“Despite this issue, recent works have introduced design patterns for system-focused protection, such as Action-Selector , Plan-then-Execute , LLM Map-Reduce , Code-Then-Execute , Dual-LLM , and Context-Minimization patterns [59].”
Just Fine-tune Twice: Selective Differential Privacy for Large Language ModelsPreprint
Shi, W., Shea, R., Chen, S., Zhang, C., Jia, R., & Yu, Z. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2204.07667 · Cited by 63 · cited 1×
“Techniques such as zero-knowledge proofs [138] or differential privacy [139] also face limitations in dealing with the dynamic and compositional nature of natural language interactions [140].”
OpenVLA: An Open-Source Vision-Language-Action Model
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62
Inputs of Coma: Static Detection of Denial-of-Service Vulnerabilities
Chang, R. M., Jiang, G.-F., Ivancic, F., Sankaranarayanan, S., & Shmatikov, V. (2009) · IEEE Computer Security Foundations Symposium · doi:10.1109/csf.2009.13 · Cited by 62 · cited 1×
“We also delineate potential shortcomings of these approaches.”
The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., & Yu, P. S. (2024) · arXiv · doi:10.48550/arxiv.2407.19354 · Cited by 60
The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., & Yu, P. S. (2026) · ACM Computing Surveys · doi:10.1145/3773080 · Cited by 58 · cited 1×
“Once input data is processed by an LLM, it is transformed into distributed latent representations, eliminating persistent identifiers that could link actions to their source [142].”
A Critical Evaluation of Defenses against Prompt Injection Attacks: [Dataset/Tool Paper]
Jia, Y., Shao, Z., Liu, Y., Jia, J., Song, D. X., & Gong, N. Z. (2026) · ACM Symposium on Access Control Models and Technologies (SACMAT) · doi:10.1145/3750555.3811884 · Cited by 58
Raza, S., Sapkota, R., Karkee, M., & Emmanouilidis, C. (2026) · AI Open · doi:10.1016/j.aiopen.2026.02.006 · Cited by 57
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.06674 · Cited by 56 · cited 2×
“For instance, Llama Guard [195] targets text-based LLMs, LlavaGuard [204] extends to image-based multimodal models, and Safewatch [205] addresses video generation.”
Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applications
Zhang, Q., Zeng, B., Zhou, C., Go, G., Shi, H., & Jiang, Y. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663786 · Cited by 56 · cited 1×
“Intentional prompt injection can also spread among agents in multi-agent systems, resulting in persistent hijacking or coordinated compromise via reliable communication channels [38].”
HuRef: HUman-REadable Fingerprint for Large Language Models
Zeng, B.-Y., Wang, L.-Z., Hu, Y.-C., Xu, Y., Zhou, C., Wang, X.-B., Yu, Y., & Lin, Z.-H. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2312.04828 · Cited by 56 · cited 1×
“Techniques such as zero-knowledge proofs [138] or differential privacy [139] also face limitations in dealing with the dynamic and compositional nature of natural language interactions [140].”
Jatmo: Prompt Injection Defense by Task-Specific Finetuning
Piet, J., Alrashed, M., Sitawarin, C., Chen, S., Wei, Z., Sun, E., Alomair, B., & Wagner, D. (2023) · arXiv · doi:10.48550/arxiv.2312.17673 · Cited by 54 · cited 1×
“By checking for invalid or unexpected responses, such as forced tokens (e.g., “HACKED”), these approaches can identify compromised generations [97, 189].”
Schulhoff, S., Pinto, J., Khan, A., Bouchard, L.-F., Si, C., Anati, S., Tagliabue, V., Kost, A., Carnahan, C., & Boyd-Graber, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.302 · Cited by 54
A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos
Yao, Y., Tong, X., Wang, R.-F., Wang, Y.-X., Li, L., Liu, L., Teng, Y., & Wang, Y.-C. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.15806 · Cited by 53
Clop, C., & Teglia, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.14479 · Cited by 52 · cited 1×
“Both direct prompt hijacking (e.g. strings such as “ ignore all previous instructions and... "), and indirect vectors (malicious payloads encoded in external sources like papers, APIs, or online content), can be intentional in nature [77, 78].”
Group-Aware Coordination Graph for Multi-Agent Reinforcement LearningPreprint
Duan, W., Lu, J., & Xuan, J. (2024) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2404.10976 · Cited by 51 · cited 1×
“We discuss these categories (and their sub-classifications) in further detail, below:”
SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent ExplanationsPreprint
Chen, Z., Pinto, F., Pan, M., & Li, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.06878 · Cited by 50
Examining Identity Drift in Conversations of LLM AgentsPreprint
Choi, J., Hong, Y., Kim, M., & Kim, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.00804 · Cited by 49 · cited 1×
“Thus, the central challenge is not only detecting explicit attacks but also designing agents that remain aligned under ambiguous or shifting input conditions, highlighting the need for stronger defenses in input validation, context management, and instruction adherence.”
AIIDA-SQL: An Adaptive Intelligent Intrusion Detector Agent for detecting SQL Injection attacks
Pinzón, C., de Paz, J. D., Bajo, J., Herrero, Á., & Corchado, E. (2010) · International Conference on Health Information Science · doi:10.1109/his.2010.5600026 · Cited by 48 · cited 1×
“Moreover, the aims of attackers in these situations can include data theft, fraud, ransomware deployment, and network lateral movement, according to various industry assessments [16].”
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
Liu, E. Z., Guu, K., Pasupat, P., Shi, T., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1802.08802 · Cited by 47 · cited 1×
“On the long-standing MiniWoB/MiniWoB++ benchmarks, proposed methods have continued to make progress [224–226].”
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.05566 · Cited by 46 · cited 1×
“…maintain safe behavior across multi-step subgoals, interruptions, and/or distribution shifts; and (ii) latent misbehavior detection : recent work [246] has shown that deceptive policies can persist through standard safety training and only activate under triggers, undermining short-episode…”
Learning to Use Tools via Cooperative and Interactive AgentsPreprint
Shi, Z., Gao, S., Chen, X., Feng, Y., Yan, L., Shi, H., Yin, D., Ren, P., Verberne, S., & Ren, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.03031 · Cited by 46 · cited 1×
“Shi et al.’s ConAgents [107] framework highlights structured cooperation among specialized agents for tool selection, execution, and action calibration, allowing agents to iteratively recover from failures and refine their actions.”
From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
Ferrag, M. A., Tihanyi, N., Hamouda, D., Maglaras, L., Lakas, A., & Debbah, M. (2026) · ICT Express · doi:10.1016/j.icte.2025.12.001 · Cited by 46 · cited 4×
“For instance, embedded backdoors are malicious triggers hidden in prompts or model parameters that misuse MCP-enabled tool access [110, 116].”
InfluentialManipulating Multimodal Agents via Cross-Modal Prompt Injection
Wang, L., Ying, Z., Zhang, T., Liang, S., Hu, S., Zhang, M., Liu, A., & Liu, X. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755211 · Cited by 45 · cited 5×
“An LLM-powered screening agent, for instance, may concatenate or jointly summarize resumes that contain fragments of a malicious prompt strewn throughout sections to manipulate the model into giving positive evaluations regardless of the actual qualifications [60, 59].”
InfluentialEmbodied AI: A Survey on the Evolution from Perceptive to Behavioral Intelligence
Yifan, C., Wei, M., Wang, X., Liu, Y., Wang, J., Song, H., Ma, L., Di, D., Sun, C., Liu, K., et al. (2025) · SmartBot · doi:10.1002/smb2.70003 · Cited by 43
Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks
Chen, Y., Hu, X., Yin, K., Li, J., & Zhang, S. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755646 · Cited by 40 · cited 1×
“For example, by converting guard requests into executable code, GuardAgent applies safety constraints during runtime without altering agents or retraining models [200].”
Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an ExamplePreprint
Chen, Y., Pesaranghader, A., Sadhu, T., & Yi, D. H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.06318 · Cited by 39
DoomArena: A framework for Testing AI Agents Against Evolving Security ThreatsPreprint
Boisvert, L., Bansal, M., Evuru, C. K. R., Huang, G., Puri, A., Bose, A., Fazel, M., Cappart, Q., Stanley, J., Lacoste, A., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.14064 · Cited by 39
Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI AgentsPreprint
Sanz-Gómez, M., Mayoral-Vilches, V., Balassone, F., Navarrete-Lozano, L. J., Chavez, C. R. J. V., & de Torres, M. D. M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.24317 · Cited by 33
Large language models (LLMs) and the institutionalization of misinformation
Garry, M., Chan, W. M., Foster, J., & Henkel, L. A. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.08.007 · Cited by 33 · cited 1×
“These agents have the potential to act in unpredictable ways, overcome human limitations, and expose users to a series of negative consequences, such as disinformation and hijacking [170, 171].”
Zong, X., Shen, Z., Wang, L., Lan, Y., & Yang, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.15163 · Cited by 32
Large Language Model Evaluation Via Multi AI Agents: Preliminary results
Rasheed, Z., Waseem, M., Systä, K., & Abrahamsson, P. (2024) · Tampere University Institutional Repository (Tampere University) · doi:10.48550/arxiv.2404.01023 · Cited by 32 · cited 1×
“On the other hand, for tasks such as code generation, obtaining one correct solution (out of many possible generations) is feasible (and quantified as the pass@ k metric [236]).”
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
(2024) · Proceedings of the 30th Conference on Pattern Languages of Programs · doi:10.64346/plop2023p05 · Cited by 32 · cited 1×
“Prompts are commands that specify how an AI agent should behave [56].”
Spoof resilient coordination for distributed multi-robot systems
Renganathan, V., & Summers, T. (2017) · International Symposium on Multi-Robot and Multi-Agent Systems · doi:10.1109/mrs.2017.8250942 · Cited by 32 · cited 1×
“…protocol-specific threats, such as fake agent advertisement and unauthorized registration , which allow adversaries to impersonate agents or take over delegated tasks [121], and recursive DoS attacks , which are triggered by repeated task delegation that causes deadlocks or unbounded loops [122].”
AudioJailbreak: Jailbreak Attacks Against End-to-End Large Audio-Language Models
Chen, G., Song, F., Zhao, Z., Jia, X., Liu, Y., Qiao, Y., Zhang, W., Tu, W., Yang, Y., & Du, B. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3661073 · Cited by 32 · cited 1×
“Because LLMs are still able to interpret and execute such hidden instructions with knowledge obtained during pre-training, obfuscation enables adversaries to circumvent naïve, pattern-based defenses [60, 97].”
Long-context LLMs Struggle with Long In-context Learning
Li, T., Zhang, G., Do, Q. D., Yue, X., & Chen, W. (2024) · arXiv · doi:10.48550/arxiv.2404.02060 · Cited by 31
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31 · cited 10×
“To bridge this gap the WebArena benchmark was proposed as a testbed for approaches that explicitly incorporate memory and feedback to improve reliability [162].”
InfluentialCLIBE: Detecting Dynamic Backdoors in Transformer-based NLP ModelsPreprint
Zeng, R., Chen, X., Pu, Y., Zhang, X., Du, T., & Ji, S.-L. (2024) · Network and Distributed System Security Symposium · doi:10.48550/arxiv.2409.01193 · Cited by 30
Seven security challenges in cross-domain multi-agent LLM systems
Ko, R., Chung, J., Zheng, S., Xiao, C., Kim, T.-W., Onizuka, M., & Shin, W.-Y. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-026-00128-9 · Cited by 29 · cited 6×
“Each domain typically enforces independent logging, retention, and auditing policies, preventing a unified trace of activity [109].”
InfluentialTowards a HIPAA Compliant Agentic AI System in HealthcarePreprint
Neupane, S., Mitra, S., Mittal, S., & Rahimi, S. (2025) · arXiv.org · doi:10.48550/arxiv.2504.17669 · Cited by 28 · cited 1×
“Healthcare AI agents [29–33] have been developed that can continuously monitor chronic conditions for patients, adapt care strategies in real-time, handle patient interactions/follow-ups, act as scribes for reducing clinician documentation burden, and accelerate drug discovery via autonomous…”
System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User InjectionPreprint
Li, Z., Guo, J., & Cai, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.06493 · Cited by 28 · cited 1×
“Thus, the central challenge is not only detecting explicit attacks but also designing agents that remain aligned under ambiguous or shifting input conditions, highlighting the need for stronger defenses in input validation, context management, and instruction adherence.”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.04249 · Cited by 28
Detecting Language Model Attacks with Perplexity
Alon, G., & Kamfonas, M. (2023) · arXiv · doi:10.48550/arxiv.2308.14132 · Cited by 27 · cited 1×
“A representative approach is the use of the perplexity metric (essentially, the exponent of the cross-entropy autoregressive language modeling loss) as a signal, where unexpectedly high values suggest unnatural or out-of-context phrasing often correlated with injected instructions [193].”
Prompt Injection Detection and Mitigation via AI Multi-Agent NLP FrameworksPreprint
Gosmar, D., Dahl, D. A., & Gosmar, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.11517 · Cited by 27 · cited 7×
“However, they struggle with more complex or subtle attacks [97].”
InfluentialLLM Agents can Autonomously Exploit One-day Vulnerabilities
Fang, R., Bindu, R., Gupta, A., & Kang, D. (2024) · arXiv · doi:10.48550/arxiv.2404.08144 · Cited by 26 · cited 3×
“Notably, GPT-4 has generally outperformed all other examined models and conventional vulnerability scanners like OWASP ZAP [102] and Metasploit [103], achieving an 87% success rate when given CVE descriptions [101].”
MIP against Agent: Malicious Image Patches Hijacking Multimodal OS AgentsPreprint
Aichberger, L., Paren, A., Li, G., Torr, P., Gal, Y., & Bibi, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.10809 · Cited by 25
Manipulating Multimodal Agents via Cross-Modal Prompt InjectionPreprint
Wang, L., Ying, Z., Zhang, T., Liang, S., Hu, S., Zhang, M., Liu, A., & Liu, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.14348 · Cited by 24 · cited 5×
“An LLM-powered screening agent, for instance, may concatenate or jointly summarize resumes that contain fragments of a malicious prompt strewn throughout sections to manipulate the model into giving positive evaluations regardless of the actual qualifications [60, 59].”
InfluentialA New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems
Wu, F., Zhang, N., Jha, S., McDaniel, P., & Xiao, C. (2024) · arXiv · doi:10.48550/arxiv.2402.18649 · Cited by 23 · cited 1×
“Vulnerabilities in sandbox implementations themselves have been documented: Wu et al. [215] found missing file isolation constraints that allowed cross-session leakage, and researchers have reported flaws in dependency installation pipelines that enabled large-scale code execution attacks [216].”
Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree
Johnson, S., Pham, V., & Le, T. Q. (2025) · arXiv.org · doi:10.48550/arxiv.2507.14799 · Cited by 22 · cited 2×
“Notably, they show how login credentials from many platforms can be collected by a single malicious trigger [74].”
Luo, Y.-X., Li, Z.-Y., Liu, J.-C., Cui, J., Zhao, X.-H., & Shen, Z.-Q. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2505.24878 · Cited by 21 · cited 1×
“Studies such as Open CaptchaWorld [167] show that even advanced multimodal agents struggle with CAPTCHAs, achieving at best a 40% success rate compared to nearly 100% for humans.”
BadJudge: Backdoor Vulnerabilities of LLM-as-a-JudgePreprint
Tong, T., Wang, F., Zhao, Z., & Chen, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.00596 · Cited by 21 · cited 1×
“(2) LLMs-as-a-judge are increasingly being used for agentic AI evaluation [238], but it is not clear how reliable or resilient the judge framework is, given that past work has shown LLM judges can suffer from critical security issues themselves [249].”
Chen, Y.-L., Li, H.-R., Sui, Y., Liu, Y., He, Y.-F., Bai, X.-L., Fei, C., Li, Y.-B., Ma, H.-Z., Song, Y.-Q., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2504.20472 · Cited by 21 · cited 1×
“For instance, users can be socially engineered by an attacker into approving unsafe actions 2 , or become overwhelmed by the complexity of long execution traces.”
“Whispering MLaaS”: Exploiting Timing Channels to Compromise User Privacy in Deep Neural Networks
Shukla, S., Alam, M., Bhattacharya, S., Mitra, P., & Mukhopadhyay, D. (2023) · IACR Transactions on Cryptographic Hardware and Embedded Systems · doi:10.46586/tches.v2023.i2.587-613 · Cited by 21 · cited 1×
“Further, confidentiality breaches are inference and side-channel timing exploits that leak sensitive contextual data exposed through MCP interfaces [119, 120].”
Identifying the Risks of LM Agents with an LM-Emulated Sandbox
Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C. J., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2309.15817 · Cited by 19
Hu, M., Chen, T., Chen, Q., Mu, Y., Shao, W., & Luo, P. (2024) · arXiv · doi:10.48550/arxiv.2408.09559 · Cited by 19 · cited 1×
“Similarly, agents struggle to manage memory appropriately over long horizons, leading to erroneous or redundant context [244].”
Improving Alignment and Robustness with Circuit Breakers
Zou, A., Phan, L., Wang, J., Duenas, D., Lin, M., Andriushchenko, M., Wang, R., Kolter, Z., Fredrikson, M., & Hendrycks, D. (2024) · arXiv · doi:10.48550/arxiv.2406.04313 · Cited by 19 · cited 1×
“Several papers investigate how to structure judges (rubrics, multi-criteria prompts), validate them against humans, and reduce hallucinated assessments [232, 233].”
Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions
Gridach, M., Nanavati, J., Abidine, K. Z. E., Mendes, L., & Mack, C. (2025) · arXiv · doi:10.48550/arxiv.2503.08979 · Cited by 18 · cited 1×
“This line of research highlights the potential of agentic AI in accelerating scientific progress and expanding the knowledge frontier [24, 25].”
MPC-Minimized Secure LLM InferencePreprint
Rathee, D., Li, D.-C., Stoica, I., Zhang, H., & Popa, R. (2024) · arXiv.org · doi:10.48550/arxiv.2408.03561 · Cited by 18 · cited 1×
“Techniques such as fully homomorphic encryption [158], multi-party computation [159], and zero-knowledge proofs [160] offer partial mitigation, but they are computationally expensive and do not fully prevent manipulation in large-scale and multi-domain agentic networks.”
Adversarial Illusions in Multi-Modal EmbeddingsPreprint
Zhang, T., Jha, R., Bagdasaryan, E., & Shmatikov, V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.11804 · Cited by 18 · cited 1×
“Other work has explored similar attacks at the embedding-level as well [91].”
Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing
Askari, H., Chhabra, A., Chen, M., & Mohapatra, P. (2024) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2406.03993 · Cited by 18 · cited 1×
“A number of these security problems in agentic AI systems compound upon the vulnerabilities and deficiencies of the base LLMs [36], while others are novel and occur due to the unique landscape of agent-agent interactions.”
Self-citationLayerIF: Estimating Layer Quality for Large Language Models using Influence Functions
Askari, H., Gupta, S., Wang, F., Chhabra, A., & Chen, M. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2505.23811 · Cited by 17 · cited 1×
“148, 149], activation tracing [150], or model-source attribution queries [151–153], provide limited visibility and often fail in these complex multi-agent settings.”
Self-citationLiu, B., Li, X., Zhang, J., Wang, J., He, T., Hong, S., Liu, H., Zhang, S., Song, K., Zhu, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.01990 · Cited by 16
PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
Zou, W., Geng, R., Wang, B., & Jia, J. (2024) · arXiv · doi:10.48550/arxiv.2402.07867 · Cited by 16 · cited 1×
“Other work on retrieval corruption and federated training manipulation, studies scenarios where adversaries distort contextual inputs in retrieval-augmented generation pipelines [117] or inject malicious updates in federated learning settings to corrupt distributed training data [118].”
LLM Agents can Autonomously Hack Websites
Fang, R., Bindu, R., Gupta, A., Zhan, Q., & Kang, D. (2024) · arXiv · doi:10.48550/arxiv.2402.06664 · Cited by 16 · cited 4×
“For instance, in [100], the authors design LLM agents capable of performing intricate multi-step website exploits via strategic combinations of tool calls and dynamic planning.”
InfluentialSandboxEval: Towards Securing Test Environment for Untrusted CodePreprint
Rabin, M. R. I., Hostetler, J., McGregor, S., Weir, B., & Judd, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.00018 · Cited by 16 · cited 2×
“By confining execution contexts, these designs prevent leakage of sensitive data and reduce the attack surface for adversarial code injection [206, 211].”
Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent SystemsPreprint
Peigné, P., Kniejski, M., Sondej, F., David, M., Hoelscher-Obermaier, J., de Witt, C. S., & Kran, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.19145 · Cited by 16 · cited 2×
“In multi-agent settings, Peigné et al. [145] show that confining agent interactions reduces exposure to infectious malicious prompts, but at the cost of collaborative efficiency.”
DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agents
Yang, S., Hu, Z.-B., Li, X.-S., Wang, C., Yu, T., Xu, X.-W., Zhu, L.-M., & Yao, L. (2025) · The Web Conference · doi:10.1145/3774904.3792688 · Cited by 16 · cited 1×
“…through interconnected agent workflows [79]; context tampering , which involves manipulation of payloads to misguide execution or leak sensitive information [110]; and memory manipulation , where malicious inputs compromise internal agent state or corrupt persistent memory of AI agents [123].”
Aligning Individual and Collective Objectives in Multi-Agent Cooperation
Li, Y., Zhang, W.-H., Wang, J.-H., Zhang, S., Du, Y.-L., Wen, Y., & Pan, W. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2402.12416 · Cited by 16 · cited 1×
“Once embedded, malicious agents can collude to form a hidden consensus, amplifying their influence until legitimate safeguards collapse [126, 127].”
Red-Teaming LLM Multi-Agent Systems via Communication Attacks
He, P., Lin, Y., Dong, S., Xu, H., Xing, Y., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2502.14847 · Cited by 15 · cited 1×
“Other work [248] has highlighted vulnerabilities in inter-agent communication protocols (such as message interception and manipulation attacks) and shown that these threats can compromise real-world agentic AI applications.”
Defeating Prompt Injections by Design
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F. (2026) · 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml68715.2026.00040 · Cited by 15 · cited 1×
“Techniques such as data attribution and control-flow extraction aim to ease the verification burden [144, 184].”
Blockagents: Towards byzantine-robust llm-based multi-agent coordination via blockchain
Chen, B., Li, G., Lin, X., Wang, Z., & Li, J. (2024) · ACM Turing Award Celebration Conference 2024 · doi:10.1145/3674399.3674445 · Cited by 15 · cited 1×
“We also delineate potential shortcomings of these approaches.”
Ontology for Blind SQL Injection
Dora, J. R., Hluchý, L., & Nemoga, K. (2023) · Computing and informatics · doi:10.31577/cai_2023_2_480 · Cited by 15 · cited 1×
“These agents carry out multi-step attacks, such as utilizing Cross-Site Scripting (XSS) and Cross-Site Request Forgery (CSRF) for XSS+CSRF chaining [104], server-side template injection (SSTI) [105], and blind SQL union injections [106].”
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024) · arXiv · doi:10.48550/arxiv.2404.13208 · Cited by 14 · cited 3×
“There are two main approaches for doing so: Prompt engineering and runtime output behavior The authors in [176] propose an instruction hierarchy, which establishes priority levels for different instruction sources so that user-provided instructions are always prioritized over potentially malicious…”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses
Liu, Y., Jia, Y., Geng, R., Jia, J., & Gong, N. Z. (2023) · arXiv · doi:10.48550/arxiv.2310.12815 · Cited by 14
AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
Wang, H., Poskitt, C. M., & Sun, J. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764546 · Cited by 14
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13 · cited 1×
“Researchers have recommended ensuring human-in-the-loop control and using structured autonomy levels to define agent capabilities and restrictions in order to reduce potential dangers [173, 174].”
Generative to Agentic AI: Survey, Conceptualization, and Challenges
Schneider, J. (2025) · arXiv · doi:10.48550/arxiv.2504.18875 · Cited by 13 · cited 1×
“Existing surveys on autonomous AI agents [49–53] mostly detail their capabilities and propose evaluation benchmarks, while others explore topics such as trust and risk management [54] without a singular focus on security, unlike our work.”
DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender AgentsPreprint
Yang, S., Hu, Z., Li, X., Wang, C., Yu, T., Xu, X., Zhu, L., & Yao, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.23804 · Cited by 12 · cited 1×
“…through interconnected agent workflows [79]; context tampering , which involves manipulation of payloads to misguide execution or leak sensitive information [110]; and memory manipulation , where malicious inputs compromise internal agent state or corrupt persistent memory of AI agents [123].”
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., & Toutanova, K. N. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1490 · Cited by 12
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Andriushchenko, M., Souly, A., Dziemian, M., Duenas, D., Lin, M., Wang, J., Hendrycks, D., Zou, A., Kolter, Z., Fredrikson, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.09024 · Cited by 11
Fully Autonomous AI Agents Should Not be Developed
Mitchell, M., Ghosh, A., Luccioni, A. S., & Pistilli, G. (2025) · arXiv · doi:10.48550/arxiv.2502.02649 · Cited by 11 · cited 14×
“For instance, Llama Guard [195] targets text-based LLMs, LlavaGuard [204] extends to image-based multimodal models, and Safewatch [205] addresses video generation.”
InfluentialApplication of Homomorphic Encryption in Medical ImagingPreprint
Dutil, F., See, A., Jorio, L. D., & Chandelier, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07768 · Cited by 11 · cited 1×
“Even in privacy-preserving deployments where inputs and outputs remain encrypted [154–156], attackers may exploit the lack of complete plaintext visibility across domains.”
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., & Berant, J. (2024) · arXiv · doi:10.48550/arxiv.2407.15711 · Cited by 11
Khan, R. M. S., Tan, Z., Yun, S., Fleming, C., & Chen, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.00218 · Cited by 11 · cited 2×
“By spoofing an agent’s identity or posing as a trusted collaborator, an attacker gains entry into workflows that would otherwise be restricted [124, 125].”
StruQ: defending against prompt injection with structured queries
Chen, S., Piet, J., Sitawarin, C., & Wagner, D. (2024) · arXiv · doi:10.48550/arxiv.2402.06363 · Cited by 10
MAGPIE: A dataset for Multi-AGent contextual PrIvacy EvaluationPreprint
Juneja, G., Albalak, A., Hua, W., & Wang, W. Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.20737 · Cited by 10
Agentic Misalignment: How LLMs Could Be Insider Threats
Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., & Troy, K. (2025) · arXiv · doi:10.48550/arxiv.2510.05179 · Cited by 10 · cited 1×
“Further, researchers at Anthropic observed that generative models, when given directive autonomy, engaged in misaligned behaviors such as blackmail or corporate espionage to fulfill goals, even when those behaviors diverged from human ethical standards [41].”
Chhabra, A., Li, B., Chen, J., Mohapatra, P., & Liu, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.03869 · Cited by 10
Self-citation
Research works printed but not linked: 70
- 10.1007/978-981-97-5656-8 · DOI
- Work, 2024, AI what do large language models ‘understand, Image, 21, 1
- 10.1007/978-3-031-54827-7_5 · DOI
- 2025, AI Agents Are Here. So Are the Threats.
- 2024, LangChain Documentation
- Richards, 2024, AutoGPT: An Autonomous GPT Experiment
- Dal Cin, 2025, Three essentials for agentic AI security, MIT Sloan Manage. Rev., 1
- 2025, Just in time? Manufacturers Turn to AI to Weather Tariff Storm
- 2025, Forget Chatbots. AI Agents are the Future
- 10.1016/j.cell.2024.09.022 · DOI · Paper page
- 2025, Inside the Automated Warehouse Where Robots Are Packing Your Groceries
- 2025, Amazon’s Delivery, Logistics Get AI Boost
- 10.1007/978-3-031-90026-6_10 · DOI · Paper page
- Micro, 2025, Preventing Zero-Click AI Threats: Insights From EchoLeak
- 2025, AI: Advent of Agents Opens New Possibilities for Attackers
- 10.52202/079017-4136 · doi:10.52202/079017-4136 · Paper page
- 10.18653/v1/2025.emnlp-main.1771 · DOI
- 2025, When Llms Autonomously Attack
- 2025, OWASP GenAI LLM01: Prompt Injection
- 10.18653/v1/2025.acl-long.890 · DOI
- 10.18653/v1/2023.emnlp-main.849 · DOI
- Park, 2025, Unveiling AI Agent Vulnerabilities Part II: Code Execution
- 2024, CVE-2024-5565: Vanna.AI Remote Code Execution Vulnerability
- Chhabra, Towards fair video summarization
- 10.1609/aaai.v34i07.6918 · DOI
- Rehberger, 2024, DeepSeek AI: From Prompt Injection To Account Takeover
- 2025, Technical Blog: Strengthening AI Agent Hijacking Evaluations
- Kennedy, 2011, Metasploit: Penetration Tester’s Guide
- Muehlberger, 2020, CSRF and XSS: Practical Examples Using Burp Suite
- 10.18653/v1/2024.findings-emnlp.624 · DOI
- 2025, Introduction To MCP. Model Context Protocol
- Surapaneni, 2025, A2A: A New Era of Agent Interoperability
- Chang, 2024, AgentNetworkProtocol (ANP) GitHub Repository
- 2024, Agent Communication Protocol (Linux Foundation)
- 10.1007/978-3-030-58951-6_24 · DOI · Paper page
- 10.18653/v1/2025.acl-long.476 · DOI
- 10.52202/079017-1421 · DOI
- 10.1109/SP46214.2022.9833644 · DOI · Paper page
- 10.14722/ndss.2025.230478 · DOI
- 2025, JFrog and Hugging Face Join Forces to Expose Malicious ML Models
- 10.1145/501978.501980 · DOI · Paper page
- 10.18653/v1/2022.emnlp-main.425 · DOI
- 10.18653/v1/2022.findings-emnlp.148 · DOI
- 10.1145/2619091 · DOI
- Singh, 2025, LLM-Based Agents: The Benefits and the Risks
- Chhabra, What data benefits my classifier? Enhancing model performance and interpretability through influence-based data selection
- 10.52202/075280-0056 · DOI
- Petrovic, 2025, Advanced Interpretability Techniques for Tracing LLM Activations
- 10.52202/079017-0664 · DOI
- de Castro, Privacy-preserving large language model inference via gpu-accelerated fully homomorphic encryption
- Lu, 2024, An efficient and extensible zero-knowledge proof framework for neural networks, Cryptol. ePrint Arch., 2024/703
- 10.1163/2210-7975_hrd-9919-2016005 · doi:10.1163/2210-7975_hrd-9919-2016005
- Cihon, Chilling autonomy: Policy enforcement for human oversight of AI agents
- 10.1063/5.0222987 · DOI
- 2024, Fine-Tuned DeBERTa-V3-Base for Prompt Injection Detection
- Chen, 2024, Aligning LLMs to be robust against prompt injection, arXiv:2410.05451 · arXiv
- Helff, Llavaguard: VLM-based safeguard for vision dataset curation and safety assessment, 8322
- 10.1609/aies.v7i1.31664 · DOI
- 2024, CloudImposer: Executing Code on Millions of Google Servers With a Single Malicious Package
- 2025, NCCoE Cyber AI Profile
- 2025, Agentic AI—Threats & Mitigations
- 10.1007/978-3-032-02130-4_2 · DOI
- 10.18653/v1/2025.findings-naacl.123 · DOI
- 10.52202/079017-1650 · doi:10.52202/079017-1650 · Paper page
- 10.18653/v1/2024.acl-long.50 · DOI · Paper page
- 10.18653/v1/2025.findings-acl.1113 · DOI
- 10.18653/v1/2025.acl-industry.75 · DOI
- 10.18653/v1/2025.acl-long.1575 · DOI
- 10.18653/v1/2025.findings-acl.332 · DOI
- 10.18653/v1/2025.findings-acl.349 · DOI
Cited by
Agentic Business Process Management: A research manifesto
Calvanese, D., Casciani, A., Giacomo, G. D., Dumas, M., Fournier, F., Kampik, T., Malfa, E. L., Limonad, L., Marrella, A., Metzger, A., et al. (2026) · Information Systems · doi:10.1016/j.is.2026.102738 · Cited by 37
“However, genAI agent security is an emerging and evolving area of research (cf. [81] for a taxonomic overview) and, accordingly, APM systems face — just like any other systems that deploy genAI agents — many security challenges that remain unsolved.”
Nageshwaran, V., & Ezekiel, S. (2026) · Electronics · doi:10.3390/electronics15122740 · Cited by 4
“The second is the development of agentic AI: software systems that wrap LLM reasoning in perception, planning, memory, tool-use, and multi-agent communication loops, enabling them to autonomously execute multi-step security tasks that previously required human analysts [22–28].”
InfluentialNageshwaran, V., & Ezekiel, S. (2026) · doi:10.20944/preprints202606.0142.v1 · Cited by 4
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 108
Krishnan, V. G., & Divya, V. (2026) · VAWKUM Transactions on Computer Sciences · doi:10.21015/vtcs.v14i2.2688 · Cited by 1
Metadata
- Authors
- Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P.
- Affiliations
- University of South Florida (US)
- Venue
- IEEE Access
- Published
- 2026
- Volume
- 14
- Pages
- 49455-49482
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W7138839765
- Cited by
- 28
- References
- 288
- Citations by year
- 2026: 28
- FWCI
- 95.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Keywords
- agentic AI, large language models, AI security, security threats, threat taxonomy, security by design, AI governance, AI agent evaluation, autonomous agents, tool use, AI safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.