Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Open accessThe DOI matches its Crossref record; 264 of the 288 references it cites were found in a registry or an index.
Abstract
Agentic AI systems powered by large language models (LLMs) and endowed with planning, tool use, memory, and autonomy, are emerging as powerful, flexible platforms for automation. Their ability to autonomously execute tasks across web, software, and physical environments creates new and amplified security risks, distinct from both traditional AI safety and conventional software security. This survey outlines a taxonomy of threats specific to agentic AI, reviews recent benchmarks and evaluation methodologies, and discusses defense strategies from both technical and governance perspectives.We synthesize current research and highlight open challenges, aiming to support the development of secure-by-design agent systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The second is the development of agentic AI: software systems that wrap LLM reasoning in perception, planning, memory, tool-use, and multi-agent communication loops, enabling them to autonomously execute multi-step security tasks that previously required human analysts”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“They introduce new attack surfaces: prompt injection via untrusted observations [88], tool hijacking, and emergent collusion in multiagent settings [22,26].”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“across the full 153-paper corpus, only five studies (≈3%)- [22,71,125] for error propagation and [26,87] for tool-use amplification-report observing a compound failure mode at all”
From Agentic AI and Large Language Models for Autonomous IoT Cybersecurity: A Systematic Survey, Taxonomy, and Research Roadmap (2026) · cites this paper as [22]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hao, X.-B., & Zhang, G.-G. (2017) · Encycl. Semantic Comput. Robotic Intell. · doi:10.1142/s2425038416300184 · Cited by 86,401 · cited 2×
“…can have across diverse domains, these systems are being increasingly employed in several real-world scenarios and applications: • Automation of complex workflows: Agentic AI systems can autonomously manage supply chain processes and adapt procurement decisions in response to dynamic data…”
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“For instance, Devin , an AI software engineer developed by Cognition Labs, has demonstrated the ability to plan, code, debug, and deploy software with minimal oversight.”
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“Analogous to defending against traditional ML/AI adversarial examples in computer vision, completely preventing prompt injection is an open problem [186].”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Modern pretraining and supervised fine-tuning paradigms on dialogue-style data, which trains the model to learn short-term instruction-response behavior (while deprioritizing long-term embodied sequential state tracking), are likely resulting in these shortcomings [165, 162].”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1×
“For instance, Devin , an AI software engineer developed by Cognition Labs, has demonstrated the ability to plan, code, debug, and deploy software with minimal oversight.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“In the subsequent section, we discuss these motivations in more detail, and delineate our contributions towards augmenting the safety and security of agentic AI frameworks.”
Robust physical-world attacks on deep learning visual classification
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00175 · Cited by 2,173
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
Athalye, A., Carlini, N., & Wagner, D. (2018) · arXiv · doi:10.48550/arxiv.1802.00420 · Cited by 2,172 · cited 1×
“More generally, in adversarial attack research in ML/AI, many defenses initially reported as effective are commonly undermined soon after their release, primarily because they lack thorough evaluation against adaptive attack strategies [182, 251, 252].”
Research works printed but not linked: 70
- 10.1007/978-981-97-5656-8 · DOI
- Work, 2024, AI what do large language models ‘understand, Image, 21, 1
- 10.1007/978-3-031-54827-7_5 · DOI
- 2025, AI Agents Are Here. So Are the Threats.
- 2024, LangChain Documentation
- Richards, 2024, AutoGPT: An Autonomous GPT Experiment
- Dal Cin, 2025, Three essentials for agentic AI security, MIT Sloan Manage. Rev., 1
- 2025, Just in time? Manufacturers Turn to AI to Weather Tariff Storm
- 2025, Forget Chatbots. AI Agents are the Future
- 10.1016/j.cell.2024.09.022 · DOI · Paper page
- 2025, Inside the Automated Warehouse Where Robots Are Packing Your Groceries
- 2025, Amazon’s Delivery, Logistics Get AI Boost
- 10.1007/978-3-031-90026-6_10 · DOI · Paper page
- Micro, 2025, Preventing Zero-Click AI Threats: Insights From EchoLeak
- 2025, AI: Advent of Agents Opens New Possibilities for Attackers
- 10.52202/079017-4136 · doi:10.52202/079017-4136 · Paper page
- 10.18653/v1/2025.emnlp-main.1771 · DOI
- 2025, When Llms Autonomously Attack
- 2025, OWASP GenAI LLM01: Prompt Injection
- 10.18653/v1/2025.acl-long.890 · DOI
- 10.18653/v1/2023.emnlp-main.849 · DOI
- Park, 2025, Unveiling AI Agent Vulnerabilities Part II: Code Execution
- 2024, CVE-2024-5565: Vanna.AI Remote Code Execution Vulnerability
- Chhabra, Towards fair video summarization
- 10.1609/aaai.v34i07.6918 · DOI
- Rehberger, 2024, DeepSeek AI: From Prompt Injection To Account Takeover
- 2025, Technical Blog: Strengthening AI Agent Hijacking Evaluations
- Kennedy, 2011, Metasploit: Penetration Tester’s Guide
- Muehlberger, 2020, CSRF and XSS: Practical Examples Using Burp Suite
- 10.18653/v1/2024.findings-emnlp.624 · DOI
- 2025, Introduction To MCP. Model Context Protocol
- Surapaneni, 2025, A2A: A New Era of Agent Interoperability
- Chang, 2024, AgentNetworkProtocol (ANP) GitHub Repository
- 2024, Agent Communication Protocol (Linux Foundation)
- 10.1007/978-3-030-58951-6_24 · DOI · Paper page
- 10.18653/v1/2025.acl-long.476 · DOI
- 10.52202/079017-1421 · DOI
- 10.1109/SP46214.2022.9833644 · DOI · Paper page
- 10.14722/ndss.2025.230478 · DOI
- 2025, JFrog and Hugging Face Join Forces to Expose Malicious ML Models
- 10.1145/501978.501980 · DOI · Paper page
- 10.18653/v1/2022.emnlp-main.425 · DOI
- 10.18653/v1/2022.findings-emnlp.148 · DOI · Paper page
- 10.1145/2619091 · DOI
- Singh, 2025, LLM-Based Agents: The Benefits and the Risks
- Chhabra, What data benefits my classifier? Enhancing model performance and interpretability through influence-based data selection
- 10.52202/075280-0056 · DOI
- Petrovic, 2025, Advanced Interpretability Techniques for Tracing LLM Activations
- 10.52202/079017-0664 · DOI
- de Castro, Privacy-preserving large language model inference via gpu-accelerated fully homomorphic encryption
- Lu, 2024, An efficient and extensible zero-knowledge proof framework for neural networks, Cryptol. ePrint Arch., 2024/703
- 10.1163/2210-7975_hrd-9919-2016005 · doi:10.1163/2210-7975_hrd-9919-2016005
- Cihon, Chilling autonomy: Policy enforcement for human oversight of AI agents
- 10.1063/5.0222987 · DOI
- 2024, Fine-Tuned DeBERTa-V3-Base for Prompt Injection Detection
- Chen, 2024, Aligning LLMs to be robust against prompt injection, arXiv:2410.05451 · arXiv
- Helff, Llavaguard: VLM-based safeguard for vision dataset curation and safety assessment, 8322
- 10.1609/aies.v7i1.31664 · DOI
- 2024, CloudImposer: Executing Code on Millions of Google Servers With a Single Malicious Package
- 2025, NCCoE Cyber AI Profile
- 2025, Agentic AI—Threats & Mitigations
- 10.1007/978-3-032-02130-4_2 · DOI
- 10.18653/v1/2025.findings-naacl.123 · DOI · Paper page
- 10.52202/079017-1650 · doi:10.52202/079017-1650 · Paper page
- 10.18653/v1/2024.acl-long.50 · DOI · Paper page
- 10.18653/v1/2025.findings-acl.1113 · DOI
- 10.18653/v1/2025.acl-industry.75 · DOI
- 10.18653/v1/2025.acl-long.1575 · DOI
- 10.18653/v1/2025.findings-acl.332 · DOI
- 10.18653/v1/2025.findings-acl.349 · DOI
Cited by
Agentic Business Process Management: A research manifesto
Calvanese, D., Casciani, A., Giacomo, G. D., Dumas, M., Fournier, F., Kampik, T., Malfa, E. L., Limonad, L., Marrella, A., Metzger, A., et al. (2026) · Information Systems · doi:10.1016/j.is.2026.102738 · Cited by 37
“However, genAI agent security is an emerging and evolving area of research (cf. [81] for a taxonomic overview) and, accordingly, APM systems face — just like any other systems that deploy genAI agents — many security challenges that remain unsolved.”
Nageshwaran, V., & Ezekiel, S. (2026) · Electronics · doi:10.3390/electronics15122740 · Cited by 4
“The second is the development of agentic AI: software systems that wrap LLM reasoning in perception, planning, memory, tool-use, and multi-agent communication loops, enabling them to autonomously execute multi-step security tasks that previously required human analysts [22–28].”
InfluentialNageshwaran, V., & Ezekiel, S. (2026) · doi:10.20944/preprints202606.0142.v1 · Cited by 4
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 108
Krishnan, V. G., & Divya, V. (2026) · VAWKUM Transactions on Computer Sciences · doi:10.21015/vtcs.v14i2.2688 · Cited by 1
The Offense–Defense Balance in the Age of Agentic Cyber Warfare
Sudha, R., & Premkumar, M. J. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0413-5.ch002
A focused survey of code agent security: Attacks, defenses, and evaluation
Hu, M., Tianxiang, W., Qu, Y., Geng, Y., & Wang, H. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108288 · Cited by 2
Simões, R. T. D. P., Larriva-Novo, X. A., Sánchez-Zas, C., Villagrá, V. A., & Marín, A. (2026) · Applied Sciences · doi:10.3390/app16167915 · Cited by 1
An Action-Centric Zero Trust Maturity Model for Agentic AI Environments
Mok, J.-H., Jo, H., & Lee, S. (2026) · Sensors · doi:10.3390/s26165205 · Cited by 1
Wang, Y., Zhai, C., Cui, J., & Tao, X. (2026) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2026.113321
Agentic AI and the reasonable duty of cybersecurity: The reasonableness paradox
El‐Khoury, M., & Abdo, J. B. (2026) · Journal of Economic Criminology · doi:10.1016/j.jeconc.2026.100245
AgentCare-Guard: A Capability-Constrained Reference Monitor for Agentic FHIR TransactionsPreprint
Kataru, A. (2026) · Research Square · doi:10.21203/rs.3.rs-10728193/v1
Analyzing Structural and Semantic Barriers to Automated Adversary Emulation in Active Directory
Ding, A., Rose, A. J., & Reith, M. G. (2026) · IEEE National Aerospace and Electronics Conference (NAECON) · doi:10.1109/naecon70028.2026.11675755
“Closing this debt also requires capturing the tacit procedural knowledge that practitioners apply without thinking about it [16], [17].”
Nizamuddin, M., & Sikder, R. (2026) · Informatics · doi:10.3390/informatics13090146
“This is important because many enterprise AI controls still focus on model output moderation, while the more serious risk may occur when an agent calls a privileged tool or acts on manipulated context [17].”
InfluentialSkillCraft: Auditing Context-Dependent Legitimation of Malicious Skills in Claude Code
Han, H., Xie, Z., & Pang, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38410-2_20
AAE: A Real-Time Monitoring and Auditing System for AI Agents
Tao, S. (2026) · International Conference on Digital Society and Artificial Intelligence (DSAI) · doi:10.1145/3833541.3833637
“This makes it difficult for users to understand what actions they have taken, how much resources they have consumed, and whether unsafe behaviors has been triggered [6, 7].”
A Threat Modeling Prioritization and Automation Framework for Composable Architectures
Popescu, L.-M., & Brad, R. (2026) · Future Internet · doi:10.3390/fi18100499
Digital Twins and Agentic AI: Applications, Risks, and Governance Frameworks
Alnoman, A. A., Khwaja, A. S., Anpalagan, A., & Woungang, I. (2026) · Future Internet · doi:10.3390/fi18100501
Temara, S. (2026) · IGI Global eBooks · doi:10.4018/979-8-2600-2257-3.ch005
Agentic AI in FinTech Digital Transformation
Sallal, M., Mariwa, S., & Ocansey, K. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9594-4.ch001
Exploring Threats Targeting Edge AI Systems
Alcaraz, C., López, J., Elicha, J., & Bountakas, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41330-7_13
Agentic AI Trustworthiness Is a Full-Stack Problem: A Framework from Hardware to Governance
Farahani, B., Rouintan, M., Vahidinia, P., & Firouzi, F. (2026) · IEEE International Conference on Omni-layer Intelligent Systems (COINS) · doi:10.1109/coins69995.2026.11715577
“Early MAS studies focused on access control and trust mechanisms [1], while recent tax-onomies examine threats such as prompt injection, cyber exploitation, protocol attacks, and emergent behaviors in agent networks [2]–[4].”
Explainable AI in Emerging Cybersecurity Domains
Ansari, S. K., Ahmad, Y., Shajahan, N., Husain, S., Appukuttan, A., & Shahzad, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1970-2.ch008
THE DUAL ROLE OF LARGE LANGUAGE MODELS IN CYBERSECURITY: THREATS, DEFENSES, AND RESEARCH DIRECTIONS
Ali, M., & Manzoor, A. R. (2026) · “ADVANCED JOURNAL OF UNIVERSITY SCIENCE AND CREATIVE INNOVATIONS” AJOU SCI · doi:10.67620/ajousci.240001
Yumuşak, S., Öztoprak, F. E., Öztoprak, K., & Tuncel, Y. K. (2026) · Applied Sciences · doi:10.3390/app162010412
Gateway-Centered Predictive and Preventive Security Framework for Agentic AI Systems
Shen, J., Wang, J., Lin, C., Shi, L., & Liong, Y. L. (2026) · Journal of Machine Intelligence and Data Science · doi:10.11159/jmids.2026.007
Rashid, M. M., Uddin, M. P., & Xiang, Y. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.101086
Aziz, M. S., Ali, M., Manzoor, A. R., Younis, M., & Khabirova, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-11434787/v1
Metadata
- Authors
- Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P.
- Affiliations
- University of South Florida (US)
- Venue
- IEEE Access
- Published
- 2026
- Volume
- 14
- Pages
- 49455-49482
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W7138839765
- Cited by
- 28
- References
- 288
- Citations by year
- 2026: 28
- FWCI
- 95.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Keywords
- agentic AI, large language models, AI security, security threats, threat taxonomy, security by design, AI governance, AI agent evaluation, autonomous agents, tool use, AI safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.