Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Open accessThe DOI matches its Crossref record; 264 of the 288 references it cites were found in a registry or an index.
Abstract
Agentic AI systems powered by large language models (LLMs) and endowed with planning, tool use, memory, and autonomy, are emerging as powerful, flexible platforms for automation. Their ability to autonomously execute tasks across web, software, and physical environments creates new and amplified security risks, distinct from both traditional AI safety and conventional software security. This survey outlines a taxonomy of threats specific to agentic AI, reviews recent benchmarks and evaluation methodologies, and discusses defense strategies from both technical and governance perspectives.We synthesize current research and highlight open challenges, aiming to support the development of secure-by-design agent systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second is the development of agentic AI: software systems that wrap LLM reasoning in perception, planning, memory, tool-use, and multi-agent communication loops, enabling them to autonomously execute multi-step security tasks that previously required human analysts”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They introduce new attack surfaces: prompt injection via untrusted observations [88], tool hijacking, and emergent collusion in multiagent settings [22,26].”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“across the full 153-paper corpus, only five studies (≈3%)- [22,71,125] for error propagation and [26,87] for tool-use amplification-report observing a compound failure mode at all”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–264 of 264
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024) · arXiv · doi:10.48550/arxiv.2406.12045 · Cited by 9 · cited 3×
“The τ - BENCH pass ∧ k metric operationalizes reliability under repeated trials with minor stochasticity [227].”
Sudo Rm -rf Agentic_securityPreprint
Lee, S., Kim, J., Park, H., Yousefpour, A., Yu, S., & Song, M. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2503.20279 · Cited by 9 · cited 1×
“Security evaluation often requires testing unsafe prompts or tool sequences; executing these against live systems is risky and irreproducible [210, 232, 239–241].”
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
Zhang, H., Huang, J., Mei, K., Yao, Y., Wang, Z., Zhan, C., Wang, H., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2410.02644 · Cited by 9 · cited 1×
“This deficiency is only compounded for security considerations [245] as security risks can surface on multi-step tasks with tool use and partial observability that otherwise might not appear for shorter tasks.”
Defending Against Indirect Prompt Injection Attacks With Spotlighting
Hines, K., Lopez, G., Hall, M., Zarfati, F., Zunger, Y., & Kiciman, E. (2024) · arXiv · doi:10.48550/arxiv.2403.14720 · Cited by 8
Tools for Verifying Neural Models' Training Data
Choi, D., Shavit, Y., & Duvenaud, D. (2023) · arXiv · doi:10.48550/arxiv.2307.00682 · Cited by 8 · cited 1×
“148, 149], activation tracing [150], or model-source attribution queries [151–153], provide limited visibility and often fail in these complex multi-agent settings.”
Online Synthesis for Runtime Enforcement of Safety in Multiagent Systems
Raju, D., Bharadwaj, S., Djeumou, F., & Topcu, U. (2019) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2021.3061900 · Cited by 8 · cited 1×
“In multi-agent systems, decentralized runtime enforcement enables individual agents to generate safety-preserving adaptations locally, sidestepping the scalability limits and information-sharing constraints inherent to centralized design-time approaches [218].”
Unraveling Indirect In-Context Learning Using Influence FunctionsPreprint
Askari, H., Gupta, S., Tong, T., Wang, F., Chhabra, A., & Chen, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.01473 · Cited by 8 · cited 1×
“148, 149], activation tracing [150], or model-source attribution queries [151–153], provide limited visibility and often fail in these complex multi-agent settings.”
Self-citationMulti-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systems
Peigné, P., Kniejski, M., Sondej, F., David, M., Hoelscher-Obermaier, J., Schroeder de Witt, C., & Kran, E. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i26.34970 · Cited by 8
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models
Yi, J., Xie, Y., Zhu, B., Kiciman, E., Sun, G., Xie, X., & Wu, F. (2023) · arXiv · doi:10.48550/arxiv.2312.14197 · Cited by 8
A Survey of Multimodal Large Language Model from A Data-centric Perspective
Bai, T., Liang, H., Wan, B., Xu, Y., Li, X., Li, S., Yang, L., Li, B., Wang, Y., Cui, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.16640 · Cited by 7 · cited 1×
“It substantially outperforms earlier systems on real-world issue resolution benchmarks, solving 13.86% of tasks autonomously compared with 1.96% by prior models [22].”
InfluentialDP-Rewrite: Towards Reproducibility and Transparency in Differentially Private Text Rewriting
Igamberdiev, T., Arnold, T., & Habernal, I. (2022) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.2208.10400 · Cited by 7 · cited 1×
“Techniques such as zero-knowledge proofs [138] or differential privacy [139] also face limitations in dealing with the dynamic and compositional nature of natural language interactions [140].”
Agent-SafetyBench: Evaluating the Safety of LLM Agents
Zhang, Z., Cui, S., Lu, Y., Zhou, J., Yang, J., Wang, H., & Huang, M. (2024) · arXiv · doi:10.48550/arxiv.2412.14470 · Cited by 6
IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems
Wu, Y., Roesner, F., Kohno, T., Zhang, N., & Iqbal, U. (2024) · arXiv · doi:10.48550/arxiv.2403.04960 · Cited by 6
Privacy Side Channels in Machine Learning Systems
Debenedetti, E., Severi, G., Carlini, N., Choquette-Choo, C. A., Jagielski, M., Nasr, M., Wallace, E., & Tramèr, F. (2023) · arXiv · doi:10.48550/arxiv.2309.05610 · Cited by 6 · cited 1×
“Further, confidentiality breaches are inference and side-channel timing exploits that leak sensitive contextual data exposed through MCP interfaces [119, 120].”
Agent-as-a-Judge: Evaluate Agents with Agents
Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.10934 · Cited by 5 · cited 2×
“(2) LLMs-as-a-judge are increasingly being used for agentic AI evaluation [238], but it is not clear how reliable or resilient the judge framework is, given that past work has shown LLM judges can suffer from critical security issues themselves [249].”
Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification
Zhang, B., Tan, Y., Shen, Y., Salem, A., Backes, M., Zannettou, S., & Zhang, Y. (2024) · arXiv · doi:10.48550/arxiv.2407.20859 · Cited by 5 · cited 1×
“Finally, agent identity misuse, such as spoofing or overprivileged agents taking unauthorized action, poses serious organizational risk [47].”
Designing and Implementing LLM Guardrails Components in Production Environments
Devino, M., Ju, E., & Caldeira, P. M. (2025) · 2025 IEEE/ACM 4th International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00010 · Cited by 5
AI-Assisted Controls Change Management for Cybersecurity in the Cloud
Tupsamudre, H., Kumar, A., Agarwal, V., Gupta, N., & Mondal, S. (2022) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21537 · Cited by 5 · cited 1×
“For instance, in cloud-hosted medical pipelines, encrypted patient scans are processed by multiple agents managed by different vendors, and intermediate outputs are never exposed [157].”
Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications
Cohen, S., Bitton, R., & Nassi, B. (2024) · arXiv · doi:10.48550/arxiv.2403.02817 · Cited by 4 · cited 4×
“It is especially important to recognize that the economics of autonomous cyber-exploitation benefit adversaries significantly [96, 38, 99].”
InfluentialBackdooring Instruction-Tuned Large Language Models with Virtual Prompt InjectionPreprint
Yan, J., Yadav, V., Li, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16888 · Cited by 4 · cited 1×
“Malicious or unverified agents can be introduced during runtime as part of task-specific teams, as evidenced by backdoor attacks [129–132].”
Schulhoff, S., Pinto, J., Khan, A., Bouchard, L.-F., Si, C., Anati, S., Tagliabue, V., Kost, A. L., Carnahan, C., & Boyd-Graber, J. (2023) · arXiv · doi:10.48550/arxiv.2311.16119 · Cited by 4 · cited 1×
“Recent work has shown that LLM agents, especially those built on top of GPT-4, are capable of autonomously exploiting real-world one-day vulnerabilities, including unpatched CVEs in Python packages, online platforms, and container management systems [101].”
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3
Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon SimulationPreprint
Sun, C.-K., Zhang, D., Zhai, C.-X., & Ji, H. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.20949 · Cited by 3
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Rawles, C., Clinckemaillie, S., Chang, Y., Waltz, J., Lau, G., Fair, M., Li, A., Bishop, W., Li, W., Campbell-Ajala, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14573 · Cited by 3
The Adaptive Arms Race: Redefining Robustness in AI Security
Tsingenopoulos, I., Rimmer, V., Preuveneers, D., Pierazzi, F., Cavallaro, L., & Joosen, W. (2025) · International Symposium on Research in Attacks, Intrusions and Defenses (RAID) · doi:10.1109/raid67961.2025.00015 · Cited by 3
Zhong, B., Liu, S., Caccamo, M., & Zamani, M. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10384154 · Cited by 3
Design Patterns for Securing LLM Agents against Prompt Injections
Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., Fischer, M., Froelicher, D., Grosse, K., Naeff, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.08837 · Cited by 2 · cited 10×
“In general, LLM performance across long-horizon tasks (i.e. those that require planning, tool-use, sequential context management, and memory interactions across long episodes) suffers in comparison to shorter-term tasks [242, 243].”
InfluentialGuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
Xiang, Z., Zheng, L., Li, Y., Hong, J., Li, Q., Xie, H., Zhang, J., Xiong, Z., Xie, C., Yang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09187 · Cited by 2 · cited 3×
“Traditional runtime enforcement mechanisms tend to be reactive, and intervene only once unsafe behavior manifests, which limits foresight under distribution shifts [201, 200, 217].”
ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning
Chen, Z., Kang, M., & Li, B. (2025) · arXiv · doi:10.48550/arxiv.2503.22738 · Cited by 2
SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents
Yin, S., Pang, X., Ding, Y., Chen, M., Bi, Y., Xiong, Y., Huang, W., Xiang, Z., Shao, J., & Chen, S. (2024) · arXiv · doi:10.48550/arxiv.2412.13178 · Cited by 2
WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks
Miyai, A., Zhao, Z., Egashira, K., Sato, A., Sunada, T., Onohara, S., Yamanishi, H., Toyooka, M., Nishina, K., Maeda, R., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01952 · Cited by 2
Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
Li, A., Zhou, Y., Raghuram, V. C., Goldstein, T., & Goldblum, M. (2025) · arXiv · doi:10.48550/arxiv.2502.08586 · Cited by 2 · cited 1×
“Other agent-specific attacks include memory poisoning , that enables stealthy manipulation over time, as agents retain and act on corrupted context [42–44] and tool misuse , such as abusing calendar or API integrations, which can trigger unintended or malicious actions [45, 46].”
Seven Security Challenges in Cross-domain Multi-agent LLM SystemsPreprint
Ko, R., Jeong, J., Zheng, S., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.23847 · Cited by 2 · cited 6×
“Each domain typically enforces independent logging, retention, and auditing policies, preventing a unified trace of activity [109].”
InfluentialNahin, S. K., Askari, H., Chen, M., & Chhabra, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.08592 · Cited by 2 · cited 1×
“Prompt injection attacks remains a critical LLM vulnerability, permitting adversaries to manipulate agent behavior through crafted inputs [37–39].”
Self-citationR-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Yuan, T., He, Z., Dong, L., Wang, Y., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.10019 · Cited by 2 · cited 2×
“Rather than elicit harmful behavior, R-Judge assesses whether LLMs (and by extension, agents) can recognize safety risks in agent trajectories, providing a complementary lens for building better judge or guard(rail) components [233].”
Datta, S., Liu, H., & Chhabra, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.20207 · Cited by 2
Self-citationDissecting Adversarial Robustness of Multimodal LM Agents
Wu, C. H., Shah, R., Koh, J. Y., Salakhutdinov, R., Fried, D., & Raghunathan, A. (2024) · arXiv · doi:10.48550/arxiv.2406.12814 · Cited by 1 · cited 2×
“Note that IPI attacks take advantage of the agent’s dependence on outside tools and information sources by enclosing harmful tasks in resources that appear to be harmless, such as databases, APIs, or web pages [68–71].”
$R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning
Kang, M., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.05557 · Cited by 1
ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
Levy, I., Wiesel, B., Marreed, S., Oved, A., Yaeli, A., Mashkif, N., & Shlomov, S. (2024) · arXiv · doi:10.48550/arxiv.2410.06703 · Cited by 1 · cited 2×
“ST-WebAgentBench is an online, enterprise-focused benchmark for testing whether web agents avoid unsafe actions (e.g., destructive operations in business systems) while pursuing goals [231].”
On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents
Huang, J.-T., Zhou, J., Jin, T., Zhou, X., Chen, Z., Wang, W., Yuan, Y., Lyu, M. R., & Sap, M. (2024) · arXiv · doi:10.48550/arxiv.2408.00989 · Cited by 1 · cited 2×
“Past work [247] has shown that even a single faulty or adversarial agent can cascade failures across different multi-agent organization topologies, and resilience varies sharply with organizational structure/hierarchy .”
EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
Liao, Z., Mo, L., Xu, C., Kang, M., Zhang, J., Xiao, C., Tian, Y., Li, B., & Sun, H. (2024) · arXiv · doi:10.48550/arxiv.2409.11295 · Cited by 1
AdvAgent: Controllable Blackbox Red-teaming on Web Agents
Xu, C., Kang, M., Zhang, J., Liao, Z., Mo, L., Yuan, M., Sun, H., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2410.17401 · Cited by 1
Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection AttacksPreprint
Chen, Y., Hu, X., Yin, K., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.13053 · Cited by 1 · cited 1×
“For example, by converting guard requests into executable code, GuardAgent applies safety constraints during runtime without altering agents or retraining models [200].”
A Critical Evaluation of Defenses against Prompt Injection AttacksPreprint
Jia, Y., Shao, Z., Liu, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.18333 · Cited by 1
The Adaptive Arms Race: Redefining Robustness in AI SecurityPreprint
Tsingenopoulos, I., Rimmer, V., Preuveneers, D., Pierazzi, F., Cavallaro, L., & Joosen, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.13435 · Cited by 1
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2401.13649 · Cited by 1
WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
Evtimov, I., Zharmagambetov, A., Grattafiori, A., Guo, C., & Chaudhuri, K. (2025) · arXiv · doi:10.48550/arxiv.2504.18575 · Cited by 1 · cited 2×
“More specifically, we cover issues associated with measuring long-horizon safety in agentic systems, approaches for multi-agent system security, and developing better benchmarks for security evaluation of AI agents, among other considerations.”
CRAB: cross-environment agent benchmark for multimodal language model agents
Xu, T., Chen, L., Wu, D.-J., Chen, Y., Zhang, Z., Yao, X., Xie, Z., Chen, Y., Liu, S., Qian, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.01511 · Cited by 1
Vitel, D., & Chhabra, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.04715 · Cited by 1
Self-citationServer-Side Template Injection with Custom Exploit
Mamtora, R., Sharma, P., & Patel, J. (2021) · doi:10.32628/ijsrset218318 · Cited by 1 · cited 1×
“These risks are not rooted in the agent’s internal reasoning or learning capabilities, but rather in the mismatch, fragility, or variability of the interfaces and environments through which the agent perceives and acts [162–164].”
Prompt Injection 2.0: Hybrid AI Threats
McHugh, J., Šekrst, K., & Cefalu, J. (2025) · arXiv · doi:10.48550/arxiv.2507.13169 · cited 6×
“…into two main types: recursive injection , where a single malicious prompt triggers a chain of compromised behaviors across future interactions [61, 38, 95], and autonomous propagation , which includes multi-agent infections and AI worms that spread malicious prompts across agents or system…”
InfluentialToolGen: Unified Tool Retrieval and Calling via Generation
Wang, R., Han, X., Ji, L., Wang, S., Baldwin, T., & Li, H. (2024) · arXiv · doi:10.48550/arxiv.2410.03439 · cited 1×
“For instance, in benchmarks like WebArena, which evaluates general-purpose web interaction tasks, GPT-4’s performance illustrates some of these challenges [162] where fine-grained actions like scrolling, hovering, and tab switching add needless complexity for the model and are frequently misused.”
Permissive Information-Flow Analysis for Large Language Models
Siddiqui, S. A., Gaonkar, R., Köpf, B., Krueger, D., Paverd, A., Salem, A., Tople, S., Wutschitz, L., Xia, M., & Zanella-Béguelin, S. (2024) · arXiv · doi:10.48550/arxiv.2410.03055 · cited 2×
“Unlike conventional software systems that can implement taint checking or explicit information flow tracking [143], these latent representations make tracing infeasible [144].”
AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language ModelsPreprint
Chen, G., Fu, S., Zhao, Z., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.14103 · cited 1×
“Because LLMs are still able to interpret and execute such hidden instructions with knowledge obtained during pre-training, obfuscation enables adversaries to circumvent naïve, pattern-based defenses [60, 97].”
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
Vijayvargiya, S., Soni, A. B., Zhou, X., Wang, Z. Z., Dziri, N., Neubig, G., & Sap, M. (2025) · arXiv · doi:10.48550/arxiv.2507.06134
OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents
Abhyankar, R., Qi, Q., & Zhang, Y. (2025) · arXiv · doi:10.48550/arxiv.2506.16042
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 1×
“For instance, Devin , an AI software engineer developed by Cognition Labs, has demonstrated the ability to plan, code, debug, and deploy software with minimal oversight.”
Baseline Defenses for Adversarial Attacks Against Aligned Language Models
Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.-Y., Goldblum, M., Saha, A., Geiping, J., & Goldstein, T. (2023) · arXiv · doi:10.48550/arxiv.2309.00614 · cited 1×
“In a similar fashion, two-stage GCG [66] trains a two-part adversarial string that is still effective after paraphrasing in order to get beyond defenses based on paraphrase detection.”
PromptArmor: Simple yet Effective Prompt Injection Defenses
Shi, T., Zhu, K., Wang, Z., Jia, Y., Cai, W., Liang, W., Wang, H., Alzahrani, H., Lu, J., Kawaguchi, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.15219 · cited 5×
“Detection-based defenses Detection-based defenses seek to detect malicious inputs or outputs without modifying the fundamental LLM. Input detection methods often rely on separate filters, such as guardrail models, that screen prompts before they reach the target system [175].”
InfluentialSecAlign: Defending Against Prompt Injection with Preference Optimization
Chen, S., Zharmagambetov, A., Mahloujifar, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.05451
The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
Lupinacci, M., Pironti, F. A., Blefari, F., Romeo, F., Arena, L., & Furfaro, A. (2025) · arXiv · doi:10.48550/arxiv.2507.06850
From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
Ferrag, M. A., Tihanyi, N., Hamouda, D., Maglaras, L., Lakas, A., & Debbah, M. (2025) · arXiv · doi:10.48550/arxiv.2506.23260
AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
Wang, H., Poskitt, C. M., & Sun, J. (2025) · arXiv · doi:10.48550/arxiv.2503.18666
ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction
Wang, H., Poskitt, C. M., Wei, J., & Sun, J. (2025) · arXiv · doi:10.48550/arxiv.2508.00500
Research works printed but not linked: 70
- 10.1007/978-981-97-5656-8 · DOI
- Work, 2024, AI what do large language models ‘understand, Image, 21, 1
- 10.1007/978-3-031-54827-7_5 · DOI
- 2025, AI Agents Are Here. So Are the Threats.
- 2024, LangChain Documentation
- Richards, 2024, AutoGPT: An Autonomous GPT Experiment
- Dal Cin, 2025, Three essentials for agentic AI security, MIT Sloan Manage. Rev., 1
- 2025, Just in time? Manufacturers Turn to AI to Weather Tariff Storm
- 2025, Forget Chatbots. AI Agents are the Future
- 10.1016/j.cell.2024.09.022 · DOI · Paper page
- 2025, Inside the Automated Warehouse Where Robots Are Packing Your Groceries
- 2025, Amazon’s Delivery, Logistics Get AI Boost
- 10.1007/978-3-031-90026-6_10 · DOI · Paper page
- Micro, 2025, Preventing Zero-Click AI Threats: Insights From EchoLeak
- 2025, AI: Advent of Agents Opens New Possibilities for Attackers
- 10.52202/079017-4136 · doi:10.52202/079017-4136 · Paper page
- 10.18653/v1/2025.emnlp-main.1771 · DOI
- 2025, When Llms Autonomously Attack
- 2025, OWASP GenAI LLM01: Prompt Injection
- 10.18653/v1/2025.acl-long.890 · DOI
- 10.18653/v1/2023.emnlp-main.849 · DOI
- Park, 2025, Unveiling AI Agent Vulnerabilities Part II: Code Execution
- 2024, CVE-2024-5565: Vanna.AI Remote Code Execution Vulnerability
- Chhabra, Towards fair video summarization
- 10.1609/aaai.v34i07.6918 · DOI
- Rehberger, 2024, DeepSeek AI: From Prompt Injection To Account Takeover
- 2025, Technical Blog: Strengthening AI Agent Hijacking Evaluations
- Kennedy, 2011, Metasploit: Penetration Tester’s Guide
- Muehlberger, 2020, CSRF and XSS: Practical Examples Using Burp Suite
- 10.18653/v1/2024.findings-emnlp.624 · DOI
- 2025, Introduction To MCP. Model Context Protocol
- Surapaneni, 2025, A2A: A New Era of Agent Interoperability
- Chang, 2024, AgentNetworkProtocol (ANP) GitHub Repository
- 2024, Agent Communication Protocol (Linux Foundation)
- 10.1007/978-3-030-58951-6_24 · DOI · Paper page
- 10.18653/v1/2025.acl-long.476 · DOI
- 10.52202/079017-1421 · DOI
- 10.1109/SP46214.2022.9833644 · DOI · Paper page
- 10.14722/ndss.2025.230478 · DOI
- 2025, JFrog and Hugging Face Join Forces to Expose Malicious ML Models
- 10.1145/501978.501980 · DOI · Paper page
- 10.18653/v1/2022.emnlp-main.425 · DOI
- 10.18653/v1/2022.findings-emnlp.148 · DOI · Paper page
- 10.1145/2619091 · DOI
- Singh, 2025, LLM-Based Agents: The Benefits and the Risks
- Chhabra, What data benefits my classifier? Enhancing model performance and interpretability through influence-based data selection
- 10.52202/075280-0056 · DOI
- Petrovic, 2025, Advanced Interpretability Techniques for Tracing LLM Activations
- 10.52202/079017-0664 · DOI
- de Castro, Privacy-preserving large language model inference via gpu-accelerated fully homomorphic encryption
- Lu, 2024, An efficient and extensible zero-knowledge proof framework for neural networks, Cryptol. ePrint Arch., 2024/703
- 10.1163/2210-7975_hrd-9919-2016005 · doi:10.1163/2210-7975_hrd-9919-2016005
- Cihon, Chilling autonomy: Policy enforcement for human oversight of AI agents
- 10.1063/5.0222987 · DOI
- 2024, Fine-Tuned DeBERTa-V3-Base for Prompt Injection Detection
- Chen, 2024, Aligning LLMs to be robust against prompt injection, arXiv:2410.05451 · arXiv
- Helff, Llavaguard: VLM-based safeguard for vision dataset curation and safety assessment, 8322
- 10.1609/aies.v7i1.31664 · DOI
- 2024, CloudImposer: Executing Code on Millions of Google Servers With a Single Malicious Package
- 2025, NCCoE Cyber AI Profile
- 2025, Agentic AI—Threats & Mitigations
- 10.1007/978-3-032-02130-4_2 · DOI
- 10.18653/v1/2025.findings-naacl.123 · DOI
- 10.52202/079017-1650 · doi:10.52202/079017-1650 · Paper page
- 10.18653/v1/2024.acl-long.50 · DOI · Paper page
- 10.18653/v1/2025.findings-acl.1113 · DOI
- 10.18653/v1/2025.acl-industry.75 · DOI
- 10.18653/v1/2025.acl-long.1575 · DOI
- 10.18653/v1/2025.findings-acl.332 · DOI
- 10.18653/v1/2025.findings-acl.349 · DOI
Cited by
Agentic Business Process Management: A research manifesto
Calvanese, D., Casciani, A., Giacomo, G. D., Dumas, M., Fournier, F., Kampik, T., Malfa, E. L., Limonad, L., Marrella, A., Metzger, A., et al. (2026) · Information Systems · doi:10.1016/j.is.2026.102738 · Cited by 37
“However, genAI agent security is an emerging and evolving area of research (cf. [81] for a taxonomic overview) and, accordingly, APM systems face — just like any other systems that deploy genAI agents — many security challenges that remain unsolved.”
Nageshwaran, V., & Ezekiel, S. (2026) · Electronics · doi:10.3390/electronics15122740 · Cited by 4
“The second is the development of agentic AI: software systems that wrap LLM reasoning in perception, planning, memory, tool-use, and multi-agent communication loops, enabling them to autonomously execute multi-step security tasks that previously required human analysts [22–28].”
InfluentialNageshwaran, V., & Ezekiel, S. (2026) · doi:10.20944/preprints202606.0142.v1 · Cited by 4
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 108
Krishnan, V. G., & Divya, V. (2026) · VAWKUM Transactions on Computer Sciences · doi:10.21015/vtcs.v14i2.2688 · Cited by 1
Metadata
- Authors
- Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P.
- Affiliations
- University of South Florida (US)
- Venue
- IEEE Access
- Published
- 2026
- Volume
- 14
- Pages
- 49455-49482
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W7138839765
- Cited by
- 28
- References
- 288
- Citations by year
- 2026: 28
- FWCI
- 95.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Keywords
- agentic AI, large language models, AI security, security threats, threat taxonomy, security by design, AI governance, AI agent evaluation, autonomous agents, tool use, AI safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.