The DOI matches its Crossref record; 131 of the 131 references it cites were found in a registry or an index.
AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways
Open accessPublished version of a preprint
Abstract
An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. AI agents, capable of perceiving user inputs, reasoning and planning tasks, and executing actions, have seen remarkable advancements in algorithm development and task performance. However, the security challenges they pose remain under-explored and unresolved. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps: unpredictability of multi-step user inputs, complexity in internal executions, variability of operational environments, and interactions with untrusted external entities. By systematically reviewing these threats, this article highlights both the progress made and the existing limitations in safeguarding AI agents. The insights provided aim to inspire further research into addressing the security threats associated with AI agents, thereby fostering the development of more robust and secure AI agent applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI Agents are typically designed as single-entity systems that perform goal-directed tasks by utilizing external tools, applying sequential reasoning, and integrating real-time information to complete well-defined functions [25,43].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, while pre-modern expert systems required manual updates to static knowledge bases, modern agents leverage emergent neural architectures to generalize across tasks [25].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI Agents are typically designed as single-entity systems that perform goal-directed tasks by utilizing external tools, applying sequential reasoning, and integrating real-time information to complete well-defined functions [24], [45].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [24]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“AI Agents often include basic mechanisms for interacting with dynamic inputs, allowing them to respond to real-time stimuli such as user requests, external API calls, or state changes in software environments [25,71].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Already, expanded attack surfaces and emergent risky behaviors have been observed in autonomous agent experiments [49,50,51].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [49]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“Due to the associative abilities of LLMs [25], frequent occurrences of pronouns and identity markers, such as gender, race, nationality, and culture in training data, can bias AI agents in processing data [59, 162].”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“Additionally, the other effective measures can be applied, such as fine-tuning AI agents specifically for indirect prompt injection [196, 204], alignment [121], and employing methods such as prompt engineering and post-training classifier-based security approaches [68].”
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 1×
“This limitation restricts their ability to retain and use extensive context from previous interactions.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 2×
“These vulnerabilities result in the action execution deviating from its intended course, leading to undesirable observations and final answers.”
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150 · cited 1×
“Tools such as GuardRails AI [51] and NeMo Guardrails [138] exemplify mechanisms that can prevent LLMs from accessing sensitive information or executing potentially harmful code.”
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 1,062
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · cited 5×
“Existing surveys on AI agents [87, 105, 160, 186, 211] primarily focus on their architectures and applications, without delving deeply into the security challenges and solutions.”
InfluentialThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03825 · Cited by 861
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838 · cited 1×
“The ultimate goal is to guide the action module in executing these subtasks.”
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 1×
“Research on this aspect is rarely mentioned in the field.”
From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy
Gupta, M., Akiri, C., Aryal, K., Parker, E., & Praharaj, L. (2023) · IEEE Access · doi:10.1109/access.2023.3300381 · Cited by 786 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
ChatGPT and the Rise of Large Language Models: The New AI-Driven Infodemic Threat in Public Health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · doi:10.2139/ssrn.4352931 · Cited by 708
ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1166120 · Cited by 708
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
A survey of inverse reinforcement learning: Challenges, methods and progress
Arora, S., & Doshi, P. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103500 · Cited by 610
Retrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
Jailbreaking Black Box Large Language Models in Twenty Queries
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2023) · arXiv · doi:10.48550/arxiv.2310.08419 · Cited by 522 · cited 4×
“The adversarial attack on the perception module includes prompt injection attacks [23, 49, 49, 130, 185, 196], indirect prompt injection attacks [23, 49, 49, 130, 185, 196] and jailbreak [15, 50, 83, 161, 178, 197].”
InfluentialHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477 · cited 1×
“Tools such as GuardRails AI [51] and NeMo Guardrails [138] exemplify mechanisms that can prevent LLMs from accessing sensitive information or executing potentially harmful code.”
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
Song, C. H., Sadler, B. M., Wu, J., Chao, W.-L., Washington, C., & Su, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00280 · Cited by 435
MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418 · cited 1×
“The complex interactions within the agent make defense more challenging.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407
Text Embeddings Reveal (Almost) As Much As Text
Morris, J. X., Kuleshov, V., Shmatikov, V., & Rush, A. M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.765 · Cited by 376
A Survey on ChatGPT: AI–Generated Contents, Challenges, and Solutions
Wang, Y., Pan, Y., Yan, M., Su, Z., & Luan, T. H. (2023) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2023.3300321 · Cited by 368
Weight Poisoning Attacks on Pretrained Models
Kurita, K., Michel, P., & Neubig, G. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.249 · Cited by 354
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2305.11747 · Cited by 340
Huang, X., Ruan, W., Huang, W., Jin, G., Dong, Y., Wu, C., Bensalem, S., Mu, R., Yi, Q., Zhao, X., et al. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10824-0 · Cited by 329
A Survey of IoT Security Based on a Layered Architecture of Sensing and Data Analysis
Mrabet, H., Belguith, S., Alhomoud, A., & Jemai, A. (2020) · Sensors · doi:10.3390/s20133625 · Cited by 314 · cited 1×
“R-Judge [198] Only limited to interaction process MLCommons [166] Limited to conversational agent user case ToolEmu [141] Only incorporating human assessment for robust bias evaluation, ensuring scalable and innovative benchmarks for fair and safe AI deployment.”
AI pitfalls and what not to do: mitigating bias in AI
Gichoya, J. W., Thomas, K., Celi, L. A., Safdar, N., Banerjee, I., Banja, J. D., Seyyed-Kalantari, L., Trivedi, H., & Purkayastha, S. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20230023 · Cited by 277
Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems
Doan, B. G., Abbasnejad, E., & Ranasinghe, D. C. (2020) · Annual Computer Security Applications Conference · doi:10.1145/3427228.3427264 · Cited by 263 · cited 1×
“These model-based backdoor defense measures mainly include eliminating triggers in poison data [33], removing backdoor-related neurons [76], or trying to recover triggers [18].”
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243
AI Deception: A Survey of Examples, Risks, and Potential Solutions
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2308.14752 · Cited by 243
Human-level play in the game of Diplomacy by combining language models with strategic reasoning
Meta Fundamental AI Research Diplomacy Team (FAIR)†, Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. (2022) · Science · doi:10.1126/science.ade9097 · Cited by 241 · cited 1×
“Meta aimed to train Cicero to be "largely honest and helpful to its speaking partners" [41].”
Identifying and Reducing Gender Bias in Word-Level Language Models
Bordia, S., & Bowman, S. R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop · doi:10.18653/v1/n19-3002 · Cited by 219 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
Geng, M., Wang, S., Dong, D., Wang, H., Li, G., Jin, Z. G., Mao, X., & Liao, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608134 · Cited by 209
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.847 · Cited by 194
Multi-step Jailbreaking Privacy Attacks on ChatGPT
Li, H., Guo, D., Fan, W., Xu, M., Huang, J., Meng, F., & Song, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.272 · Cited by 187
Survey of Vector Database Management Systems
Pan, J. J., Wang, J., & Li, G. (2023) · arXiv · doi:10.48550/arxiv.2310.14021 · Cited by 184
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Mandi, Z., Jain, S., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.04738 · Cited by 180
Toxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178
Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs
Kolouri, S., Saha, A., Pirsiavash, H., & Hoffmann, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00038 · Cited by 175 · cited 1×
“Publication date: June 2018. grammar (CFG) into a pushdown automaton (PDA) and restricts the language model (LLM) to only select valid actions defined by the PDA at its current state, thereby ensuring that the constraints are met in the final generated plan.”
Information Leakage in Embedding Models
Song, C., & Raghunathan, A. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3372297.3417270 · Cited by 175
Survey of vector database management systems
Pan, J. J., Wang, J., & Li, G. (2024) · The VLDB Journal · doi:10.1007/s00778-024-00864-x · Cited by 173
Probing Toxic Content in Large Pre-Trained Language Models
Ousidhoum, N. D., Zhao, X., Fang, T., Song, Y., & Yeung, D. (2021) · doi:10.18653/v1/2021.acl-long.329 · Cited by 169 · cited 1×
“, and some unsafe data can also be mixed in. Influenced by such unsafe data, AI agents can still generate unreal, toxic, biased, or even illegal content [9, 45, 53, 64, 99, 120, 146, 167, 170, 171].”
Li, H., Xu, M., & Song, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.881 · Cited by 160
The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)
Zeng, S., Zhang, J., He, P., Liu, Y., Xing, Y., Xu, H., Ren, J., Chang, Y., Wang, S., Yin, D., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.267 · Cited by 149
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
Yao, D., Zhang, J., Harris, I. G., & Carlsson, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448041 · Cited by 148
TrustAgent: Towards Safe and Trustworthy LLM-based Agents
Hua, W., Yang, X., Jin, M., Li, Z., Cheng, W., Tang, R., & Zhang, Y. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.585 · Cited by 148
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.624 · Cited by 147
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · arXiv · doi:10.48550/arxiv.2403.02691 · Cited by 143 · cited 2×
“For example, web data retrieved by AI agents using web plugins could be misinterpreted as user instructions, resulting in extraction of historical conversations, insertion of phishing links, theft of GitHub code [204], or transmission of sensitive information to attackers [185].”
MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24188 · Cited by 137
Bao, Y., Gong, W., & Yang, K. (2023) · Systems · doi:10.3390/systems11090442 · Cited by 127 · cited 1×
“To address these challenges, concurrent solutions are divided into two categories, extending LLM context window [32] and compressing historical in-context contents [47, 65, 95, 118].”
When Software Security Meets Large Language Models: A Survey
Zhu, X., Zhou, W., Han, Q., Ma, W., Wen, S., & Xiang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124971 · Cited by 127
Self-citationThe Security of Using Large Language Models: A Survey with Emphasis on ChatGPT
Zhou, W., Zhu, X., Han, Q., Li, L., Chen, X., Wen, S., & Xiang, Y. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124983 · Cited by 126 · cited 1×
“Reasoning refers to a large language model designed to analyze and deduce information, helping to draw logical conclusions from given prompts.”
Self-citationSecurity and privacy for innovative automotive applications: A survey
Le, V. H., den Hartog, J., & Zannone, N. (2018) · Computer Communications · doi:10.1016/j.comcom.2018.09.010 · Cited by 116 · cited 1×
“To address the challenges in AI agent tool interactions, a promising future direction is the implementation of strict tool use auditing.”
S3: Social-network Simulation System with Large Language Model-Empowered Agents
Gao, C., Lan, X., Lu, Z., Mao, J., Piao, J., Wang, H., Jin, D., & Li, Y. (2023) · doi:10.2139/ssrn.4607026 · Cited by 113
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
Ignore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110 · cited 6×
“The brain is primarily composed of a large language model, which is the core of an AI agent.”
InfluentialDecodingTrust: A comprehensive assessment of trustworthiness in GPT models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1361 · Cited by 107
Security, privacy, and healthcare-related conversational agents: a scoping review
May, R., & Denecke, K. (2021) · Informatics for Health and Social Care · doi:10.1080/17538157.2021.1983578 · Cited by 104 · cited 2×
“Cohen et al. [23] uses an adversarial self-replicating prompt as the worm to poison the database of a RAG-based application, extracting user private information from the AI agent ecosystem by query process.”
Language Models as Agent Models
Andreas, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.423 · Cited by 92
MarioGPT: Open-Ended Text2Level Generation through Large Language Models
Sudhakaran, S., González-Duque, M., Freiberger, M., Glanois, C., Najarro, E., & Risi, S. (2023) · neural information processing systems · doi:10.52202/075280-2361 · Cited by 90
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81 · cited 1×
“ToolEmu [141] identifies some failures of AI agents since the action execution requires excessive tool permissions, leading to the execution of highly risky commands without user permission.”
ChatDev: Communicative Agents for Software Development
Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.07924 · Cited by 81
Methods for Measuring, Updating, and Visualizing Factual Beliefs in Language Models
Hase, P., Diab, M., Çelikyılmaz, A., Li, X., Kozareva, Z., Stoyanov, V., Bansal, M., & Iyer, S. V. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.199 · Cited by 80 · cited 1×
“One is the threat during the communication process between the agent and the tool ( i.e. , occurring in the input, observation, and final answer), termed Agent2Tool threats .”
Do large language models show decision heuristics similar to humans? A case study using GPT-3.5.
Suri, G., Slater, L. R., Ziaee, A., & Nguyen, M. (2024) · Journal of Experimental Psychology: General · doi:10.1037/xge0001547 · Cited by 79
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76
PrivacyAsst: Safeguarding User Privacy in Tool-Using Large Language Model Agents
Zhang, X., Xu, H., Ba, Z., Wang, Z., Hong, Y., Liu, J., Qin, Z., & Ren, K. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3372777 · Cited by 75 · cited 3×
“PrivacyAsst[207] Failed to address identity disclosure, and also has 1100x extra computation cost compared with vanilla AI agent.”
InfluentialContrastive Learning Reduces Hallucination in Conversations
Sun, W., Shi, Z., Gao, S., Ren, P., De Rijke, M., & Ren, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26596 · Cited by 69
Factuality Enhanced Language Models for Open-Ended Text Generation
Lee, N., Ping, W., Xu, P., Patwary, M., Fung, P., Shoeybi, M., & Catanzaro, B. (2022) · arXiv · doi:10.48550/arxiv.2206.04624 · Cited by 68 · cited 1×
“As the model scales up, hallucinations also become more severe [55, 79].”
(2022) · doi:10.52202/068431-2506 · Cited by 68
Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
Ulmer, D., Mansimov, E., Lin, K., Sun, L., Gao, X., & Zhang, Y. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.566 · Cited by 60
A Brief Survey of Vector Databases
Xie, X., Liu, H., Hou, W., & Huang, H. (2023) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia60676.2023.10429609 · Cited by 58
Bukharin, A., Li, Y., Yu, Y., Zhang, Q., Chen, Z.-H., Zuo, S.-M., Zhang, C., Zhang, S.-A., & Zhao, T. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.10810 · Cited by 57 · cited 1×
“Cohen et al. [23] uses an adversarial self-replicating prompt as the worm to poison the database of a RAG-based application, extracting user private information from the AI agent ecosystem by query process.”
Spatiotemporal Attacks for Embodied Agents
Liu, A., Huang, T., Liu, X., Xu, Y., Ma, Y., Chen, X., Maybank, S. J., & Tao, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_8 · Cited by 56 · cited 3×
“Additionally, the other effective measures can be applied, such as fine-tuning AI agents specifically for indirect prompt injection [196, 204], alignment [121], and employing methods such as prompt engineering and post-training classifier-based security approaches [68].”
From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and PrivacyPreprint
Gupta, M., Akiri, C., Aryal, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.00691 · Cited by 56 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
Conversational health agents: a personalized large language model-powered agent framework
Abbasian, M., Azimi, I., Rahmani, A. M., & Jain, R. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf067 · Cited by 52
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Yuan, T., He, Z., Dong, L., Wang, Y., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.79 · Cited by 45
POSTER: Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications
Jiang, F., Xu, Z., Niu, L., Wang, B., Jia, J., Li, B., & Poovendran, R. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3659433 · Cited by 41
DERA: Enhancing large language model completions with dialog-enabled resolving agents
Nair, V., Schumacher, E., Tso, G., & Kannan, A. (2024) · Proceedings of the 6th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2024.clinicalnlp-1.12 · Cited by 38
CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agents
Zhao, Q., Wang, J., Zhang, Y., Jin, Y., Zhu, K., Chen, H., & Xie, X. (2023) · arXiv · doi:10.48550/arxiv.2310.17512 · Cited by 37
The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey
Masterman, T., Besen, S., Sawtell, M., & Chao, A. (2024) · arXiv · doi:10.48550/arxiv.2404.11584 · Cited by 35 · cited 3×
“The literature [105] summarized the current AI agent architecture.”
Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free
Chen, T., Zhang, Z., Zhang, Y., Chang, S., Liu, S., & Wang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00068 · Cited by 31 · cited 1×
“These model-based backdoor defense measures mainly include eliminating triggers in poison data [33], removing backdoor-related neurons [76], or trying to recover triggers [18].”
MISGENDERED: Limits of Large Language Models in Understanding Pronouns
Hossain, T., Dev, S., & Singh, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.293 · Cited by 31
The “Code” of Ethics: A Holistic Audit of AI Code Generators
Ma, W., Song, Y., Xue, M., Wen, S., & Xiang, Y. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3367737 · Cited by 31
Self-citationPreventing generation of verbatim memorization in language models gives a false sense of privacy
Ippolito, D., Tramer, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette Choo, C., & Carlini, N. (2023) · Proceedings of the 16th International Natural Language Generation Conference · doi:10.18653/v1/2023.inlg-main.3 · Cited by 28 · cited 2×
“To better explain threats in the brain module, we first show the traditional structure of the brain.”
Simple synthetic data reduces sycophancy in large language models
Wei, J., Huang, D., Lu, Y., Zhou, D., & Le, Q. V. (2023) · arXiv · doi:10.48550/arxiv.2308.03958 · Cited by 28
Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
Yang, W., Bi, X., Lin, Y., Chen, S., Zhou, J., & Sun, X. (2024) · arXiv · doi:10.48550/arxiv.2402.11208 · Cited by 26 · cited 4×
“Additionally, the other effective measures can be applied, such as fine-tuning AI agents specifically for indirect prompt injection [196, 204], alignment [121], and employing methods such as prompt engineering and post-training classifier-based security approaches [68].”
InfluentialMarioGPT: Open-Ended Text2Level Generation through Large Language ModelsPreprint
Sudhakaran, S., González-Duque, M., Glanois, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.05981 · Cited by 25 · cited 1×
“Utilizing these APIs may increase its vulnerability to attacks, and the agent can be impacted by misinformation in the observations and final answer, which we refer to as Supply Chain threats .”
A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems
Wu, F., Zhang, N., Jha, S., McDaniel, P., & Xiao, C. (2024) · arXiv · doi:10.48550/arxiv.2402.18649 · Cited by 23 · cited 6×
“[185] perception malicioususer manipulatingoutput/maliciousbehavior Chanetal[14] perception malicioususer violatedresponses Liu etal.”
InfluentialDo Large Language Models Show Decision Heuristics Similar to Humans? A Case Study Using GPT-3.5Preprint
Suri, G., Slater, L. R., Ziaee, A., & Nguyen, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4440608 · Cited by 20
Bukharin, A., Li, Y., Yu, Y., Zhang, Q., Chen, Z., Zuo, S., Zhang, C., Zhang, S., & Zhao, T. (2023) · neural information processing systems · doi:10.52202/075280-2979 · Cited by 20
The future of cognitive strategy-enhanced persuasive dialogue agents: new perspectives and trends
Chen, M., Guo, B., Wang, H., Li, H., Zhao, Q., Liu, J., Ding, Y., Pan, Y., & Yu, Z. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40057-x · Cited by 17 · cited 2×
“Future research needs to address these limitations by enhancing the accuracy and efficiency of inspection mechanisms, ensuring that they can be effectively deployed in real-world applications.”
Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learning
Liu, G., & Lai, L. (2023) · neural information processing systems · doi:10.52202/075280-1062 · Cited by 17
A First Look at Toxicity Injection Attacks on Open-domain Chatbots
Weeks, C., Cheruvu, A., Abdullah, S. M., Kanchi, S., Yao, D. D., & Viswanath, B. (2023) · Annual Computer Security Applications Conference · doi:10.1145/3627106.3627122 · Cited by 16 · cited 2×
“To defend against indirect prompt attacks, developers can impose explicit constraints on the interaction between AI agents and external resources to prevent AI agents from executing external malicious data [185].”
Annual Computer Security Applications Conference
(2023) · doi:10.1145/3627106 · Cited by 14
Detection and Defense Against Prominent Attacks on Preconditioned LLM-Integrated Virtual Assistants
Chan, C. F., Yip, D. W., & Esmradi, A. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487759 · Cited by 14 · cited 3×
“Additionally, the other effective measures can be applied, such as fine-tuning AI agents specifically for indirect prompt injection [196, 204], alignment [121], and employing methods such as prompt engineering and post-training classifier-based security approaches [68].”
Toxicity Detection with Generative Prompt-based Inference
Wang, Y.-S., & Chang, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.12390 · Cited by 13 · cited 3×
“…malicioususer dataleakage Planning Threat [71] Hallucination [16, 24, 35, 150, 181] Misalignment[8,91,131] Training Data Misalignment [30,43,59,172] Human-AgentMisalignment [8,129,131,175] Misalignment in Em-bodied Environments [36, 141, 159] Backdoor Attack [34, 192] Perception(§3.1)…”
InfluentialPager Explosion: Cybersecurity Insights and Afterthoughts
Sheng, C., Ma, W., Han, Q., Zhou, W., Zhu, X., Wen, S., Xiang, Y., & Wang, F. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.125034 · Cited by 12
Self-citationMoskal, S., Laney, S., Hemberg, E., & O'Reilly, U.-M. (2023) · arXiv · doi:10.48550/arxiv.2310.06936 · Cited by 11 · cited 1×
“These security challenges stem from four knowledge gaps that, if unaddressed, can lead to vulnerabilities [27, 97, 112, 192] and potential misuse [132].”
When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour
Ranaldi, L., & Pucci, G. (2023) · arXiv · doi:10.48550/arxiv.2311.09410 · Cited by 11 · cited 1×
“To better understand the action threats, we present the action structure as follows:”
Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learning
Wu, Y., McMahan, J., Zhu, X., & Xie, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26240 · Cited by 8
Efficient Adversarial Attacks on Online Multi-agent Reinforcement LearningPreprint
Liu, G., & Lai, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.07670 · Cited by 8
Defending Against Indirect Prompt Injection Attacks With Spotlighting
Hines, K., Lopez, G., Hall, M., Zarfati, F., Zunger, Y., & Kiciman, E. (2024) · arXiv · doi:10.48550/arxiv.2403.14720 · Cited by 8 · cited 2×
“In this part, we divide these security threats in the memory interaction into two subgroups, short-term memory interaction threats and long-term memory interaction threats.”
Effective Prompt Extraction from Language Models
Zhang, Y., Carlini, N., & Ippolito, D. (2023) · arXiv · doi:10.48550/arxiv.2307.06865 · Cited by 8 · cited 3×
“To better explain threats in the brain module, we first show the traditional structure of the brain.”
Windridge, D., Svensson, H., & Thill, S. (2020) · Adaptive Behavior · doi:10.1177/1059712319896489 · Cited by 7 · cited 3×
“Additionally, the other effective measures can be applied, such as fine-tuning AI agents specifically for indirect prompt injection [196, 204], alignment [121], and employing methods such as prompt engineering and post-training classifier-based security approaches [68].”
Optimal Cost Constrained Adversarial Attacks for Multiple Agent Systems
Lu, Z., Liu, G., Lai, L., & Xu, W. (2024) · Conference on Information Sciences and Systems · doi:10.1109/ciss59072.2024.10480189 · Cited by 5 · cited 3×
“[141] action malicioustools manipulatingoutput/maliciousbehavior Park etal.”
Introducing v0.5 of the AI Safety Benchmark from MLCommons
Vidgen, B., Agrawal, A., Ahmed, A. M., Akinwande, V., Al-Nuaimi, N., Alfaraj, N., Alhajjar, E., Aroyo, L., Bavalatti, T., Bartolo, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.12241 · Cited by 5 · cited 2×
“The MLCommons group [166] proposes a principled approach to define and construct benchmarks, which is limited to a single use case: an adult conversing with a general-purpose assistant in English.”
Putla, H., Chanakya, P., Singh, K. P., & Nagabhushan, P. N. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11625-w · Cited by 4
Camouflage Adversarial Attacks on Multiple Agent Systems
Lu, Z., Liu, G., Lai, L., & Xu, W. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619346 · Cited by 4 · cited 3×
“[123] agent2environment malicioususer manipulatingoutput/maliciousbehavior Chen etal.”
RL-Based Sim2Real Enhancements for Autonomous Beach-Cleaning Agents
Quiroga, F., Hermosilla, G., Varas, G., Alonso, F., & Schröder, K. (2024) · Applied Sciences · doi:10.3390/app14114602 · Cited by 3 · cited 2×
“We name the combination of perception, brain, and action as Intra-execution in this survey.”
Talking template injection attacks
McVey, T. (2022) · Network Security · doi:10.12968/s1353-4858(22)70067-0 · Cited by 1
Detecting Tabnabbing Attacks Via An RL-Based Agent
Fonseka, A., Pashenna, P., & Ariyadasa, S. N. (2023) · International Conference on Information Technology Research (ICITR) · doi:10.1109/icitr61062.2023.10382917 · Cited by 1 · cited 4×
“We name the combination of perception, brain, and action as Intra-execution in this survey.”
Influential
Research works printed but not linked: 114
- Mahyar Abbasian Iman Azimi Amir M. Rahmani and Ramesh Jain. 2023. Conversational health agents: A personalized LLM-powered agent framework. Retrieved from https://arxiv.org/abs/2310.02374 · arXiv
- Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. GPT-4 technical report. Retrieved from https://arxiv.org/pdf/2303.08774 · arXiv
- Divyansh Agarwal Alexander R. Fabbri Philippe Laban Shafiq Joty Caiming Xiong and Chien-Sheng Wu. 2024. Investigating the prompt leakage effect and black-box defenses for multi-turn LLM interactions. Retrieved from https://arxiv.org/pdf/2404.16251v1 · arXiv
- Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. Retrieved from https://arxiv.org/pdf/2308.14132 · arXiv
- Anthropic. 2024. Many-shot Jailbreaking. Retrieved from https://www.anthropic.com/research/many-shot-jailbreaking
- Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen Nova DasSarma Dawn Drain Stanislav Fort Deep Ganguli Tom Henighan et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. Retrieved from https://arxiv.org/pdf/2204.05862 · arXiv
- Rishabh Bhardwaj and Soujanya Poria. 2023. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. Retrieved from https://arxiv.org/pdf/2310.14303 · arXiv
- Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive environments with online reinforcement learning. In Proceedings of the ICML.
- Dake Chen Hanbin Wang Yunhao Huo Yuzhao Li and Haoyang Zhang. 2023. Gamegpt: Multi-agent collaborative framework for game development. Retrieved from https://arxiv.org/pdf/2310.08067 · arXiv
- Xuan Chen Yuzhou Nie Lu Yan Yunshu Mao Wenbo Guo and Xiangyu Zhang. 2024. RL-JACK: Reinforcement learning-powered black-box jailbreaking attack against LLMs. Retrieved from https://arxiv.org/pdf/2406.08725 · arXiv
- Yongchao Chen Jacob Arkin Yang Zhang Nicholas Roy and Chuchu Fan. 2023. Scalable multi-robot collaboration with large language models: Centralized or decentralized systems? Retrieved from https://arxiv.org/html/2309.15943v2
- Zhaorun Chen Zhen Xiang Chaowei Xiao Dawn Song and Bo Li. 2024. Agentpoison: Red-teaming LLM agents via poisoning memory or knowledge bases. Retrieved from https://arxiv.org/pdf/2407.12784 · arXiv
- Steffi Chern Zhen Fan and Andy Liu. 2024. Combating adversarial attacks with multi-agent debate. Retrieved from https://arxiv.org/pdf/2401.05998 · arXiv
- Stav Cohen Ron Bitton and Ben Nassi. 2024. Here comes the AI worm: Unleashing zero-click worms that target GenAI-powered applications. Retrieved from https://arxiv.org/pdf/2403.02817 · arXiv
- Maxwell Crouse Ibrahim Abdelaziz Kinjal Basu Soham Dan Sadhana Kumaravel Achille Fokoue Pavan Kapanipathi and Luis Lastras. 2023. Formally specifying the high-level behavior of LLM-based agents. Retrieved from https://arxiv.org/pdf/2310.08535 · arXiv
- Chenhang Cui Gelei Deng An Zhang Jingnan Zheng Yicong Li Lianli Gao Tianwei Zhang and Tat-Seng Chua. 2024. Safe+ Safe= Unsafe? Exploring how safe images can be exploited to jailbreak large vision-language models. Retrieved from https://arxiv.org/pdf/2411.11496 · arXiv
- Tianyu Cui Yanling Wang Chuanpu Fu Yong Xiao Sijia Li Xinhao Deng Yunpeng Liu Qinglin Zhang Ziyi Qiu Peiyang Li et al. 2024. Risk taxonomy mitigation and assessment benchmarks of large language model systems. Retrieved from https://arxiv.org/pdf/2401.05778 · arXiv
- Yiran Ding Li Lyna Zhang Chengruidong Zhang Yuanyuan Xu Ning Shang Jiahang Xu Fan Yang and Mao Yang. 2024. LongRoPE: Extending LLM context window beyond 2 million tokens. Retrieved from https://arxiv.org/pdf/2402.13753 · arXiv
- Tian Dong Guoxing Chen Shaofeng Li Minhui Xue Rayne Holland Yan Meng Zhen Liu and Haojin Zhu. 2023. Unleashing cheapfakes through trojan plugins of large language models. Retrieved from https://arxiv.org/pdf/2312.00374v1 · arXiv
- Yingkai Dong Zheng Li Xiangtao Meng Ning Yu and Shanqing Guo. 2024. Jailbreaking text-to-image models with LLM-based agents. Retrieved from https://arxiv.org/pdf/2408.00523 · arXiv
- Yilun Du Shuang Li Antonio Torralba Joshua B. Tenenbaum and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. Retrieved from https://arxiv.org/pdf/2305.14325 · arXiv
- Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. 2023. Guiding pretraining in reinforcement learning with large language models. In Proceedings of the ICML. PMLR.
- Eva Eigner and Thorsten Händler. 2024. Determinants of LLM-assisted decision-making. Retrieved from https://arxiv.org/pdf/2402.17385 · arXiv
- Aysan Esmradi, Daniel Wankit Yip, and Chun Fai Chan. 2023. A comprehensive survey of attack techniques, implementation, and mitigation strategies in large language models. In Proceedings of the UbiSec.
- Isabel O. Gallegos Ryan A. Rossi Joe Barrow Md Mehrab Tanjim Sungchul Kim Franck Dernoncourt Tong Yu Ruiyi Zhang and Nesreen K. Ahmed. 2023. Bias and fairness in large language models: A survey. Retrieved from https://arxiv.org/pdf/2309.00770 · arXiv
- Jonas Geiping Alex Stein Manli Shu Khalid Saifullah Yuxin Wen and Tom Goldstein. 2024. Coercing LLMs to do and reveal (almost) anything. Retrieved from https://arxiv.org/abs/2402.14020 · arXiv
- Kai Greshake Sahar Abdelnabi Shailesh Mishra Christoph Endres Thorsten Holz and Mario Fritz. 2023. More than you’ve asked for: A comprehensive analysis of novel prompt injection threats to application-integrated large language models. Retrieved from https://arxiv.org/pdf/2302.12173 · arXiv
- Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In Proceedings of the AISec. 79–90.
- Xiangming Gu Xiaosen Zheng Tianyu Pang Chao Du Qian Liu Ye Wang Jing Jiang and Min Lin. 2024. Agent smith: A single image can jailbreak one million multimodal LLM agents exponentially fast. Retrieved from https://arxiv.org/pdf/2402.08567 · arXiv
- Michael Guastalla, Yiyi Li, Arvin Hekmati, and Bhaskar Krishnamachari. 2023. Application of large language models to DDoS attack detection. In Proceedings of the SmartSP.
- Rui Hao Linmei Hu Weijian Qi Qingliu Wu Yirui Zhang and Liqiang Nie. 2023. ChatLLM network: More brains more intelligence. Retrieved from https://arxiv.org/pdf/2304.12998 · arXiv
- Kostas Hatalis, Despina Christou, Joshua Myers, Steven Jones, Keith Lambert, Adam Amos-Binks, Zohreh Dannenhauer, and Dustin Dannenhauer. 2023. Memory matters: The need to improve long-term memory in LLM-Agents. In Proceedings of the AAAI.
- Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In Proceedings of the ICLR.
- Yuki Hou, Haruki Tamoto, and Homei Miyashita. 2024. “My agent understands me better”: Integrating dynamic human-like memory recall and consolidation in LLM-based agents. In Proceedings of the CHI.
- Bin Hu Chenyang Zhao Pu Zhang Zihao Zhou Yuanhang Yang Zenglin Xu and Bin Liu. 2023. Enabling intelligent interactions between an agent and an LLM: A reinforcement learning approach. Retrieved from https://arxiv.org/pdf/2306.03604 · arXiv
- Xijie Huang Li Lyna Zhang Kwang-Ting Cheng and Mao Yang. 2023. Boosting LLM reasoning: Push the limits of few-shot learning with reinforced in-context pruning. Retrieved from https://arxiv.org/pdf/2312.08901 · arXiv
- Yue Huang Qihui Zhang Lichao Sun et al. 2023. Trustgpt: A benchmark for trustworthy and responsible large language models. Retrieved from https://arxiv.org/pdf/2306.11507 · arXiv
- S. Humeau, K. Shuster, M. Lachaux, and J. Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv. In Proceedings of the ICLR.
- Neel Jain Avi Schwarzschild Yuxin Wen Gowthami Somepalli John Kirchenbauer Ping-yeh Chiang Micah Goldblum Aniruddha Saha Jonas Geiping and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. Retrieved from https://arxiv.org/pdf/2309.00614 · arXiv
- Zhenlan Ji Daoyuan Wu Pingchuan Ma Zongjie Li and Shuai Wang. 2024. Testing and understanding erroneous planning in LLM agents through synthesized user inputs. Retrieved from https://arxiv.org/pdf/2404.17833 · arXiv
- Mintong Kang Nezihe Merve Gürel Ning Yu Dawn Song and Bo Li. 2024. C-RAG: Certified generation risks for retrieval-augmented language models. Retrieved from https://arxiv.org/pdf/2402.03181 · arXiv
- Aounon Kumar Chirag Agarwal Suraj Srinivas Soheil Feizi and Hima Lakkaraju. 2024. Certifying LLM safety against adversarial prompting. Retrieved from https://arxiv.org/pdf/2309.02705 · arXiv
- Patrick Levi and Christoph P. Neumann. 2024. Vocabulary attack to hijack large language model applications. Retrieved from https://arxiv.org/pdf/2404.02637 · arXiv
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the NeurIPS.
- Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative agents for ”Mind” exploration of large language model society. In Proceedings of the NeurIPS.
- Yuanchun Li Hao Wen Weijun Wang Xiangyu Li Yizhen Yuan Guohong Liu Jiacheng Liu Wenxing Xu Xiang Wang Yi Sun et al. 2024. Personal LLM agents: Insights and survey about the capability efficiency and security. Retrieved from https://arxiv.org/pdf/2401.05459 · arXiv
- Zelong Li Wenyue Hua Hao Wang He Zhu and Yongfeng Zhang. 2024. Formal-LLM: Integrating formal language and natural language for controllable LLM-based agents. Retrieved from https://arxiv.org/pdf/2402.00798 · arXiv
- Zeyi Liao Lingbo Mo Chejian Xu Mintong Kang Jiawei Zhang Chaowei Xiao Yuan Tian Bo Li and Huan Sun. 2024. Eia: Environmental injection attack on generalist web agents for privacy leakage. Retrieved from https://arxiv.org/pdf/2409.11295 · arXiv
- Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023. AvalonBench: Evaluating LLMs playing the game of avalon. In Proceedings of the NeurIPS Workshop.
- Baihan Lin Djallel Bouneffouf Guillermo Cecchi and Kush R. Varshney. 2023. Towards healthy AI: large language models need therapists too. Retrieved from https://arxiv.org/pdf/2304.00416 · arXiv
- Jiaju Lin Haoran Zhao Aochi Zhang Yiting Wu Huqiuyue Ping and Qin Chen. 2023. Agentsims: An open-source sandbox for large language model evaluation. Retrieved from https://arxiv.org/pdf/2308.04026 · arXiv
- Sheng Liu Lei Xing and James Zou. 2023. In-context vectors: Making in context learning more effective and controllable through latent space steering. Retrieved from https://arxiv.org/pdf/2311.06668 · arXiv
- Tong Liu Zizhuang Deng Guozhu Meng Yuekang Li and Kai Chen. 2023. Demystifying rce vulnerabilities in LLM-integrated apps. Retrieved from https://arxiv.org/pdf/2309.02926 · arXiv
- Yi Liu Gelei Deng Yuekang Li Kailong Wang Tianwei Zhang Yepang Liu Haoyu Wang Yan Zheng and Yang Liu. 2023. Prompt injection attack against LLM-integrated applications. Retrieved from https://arxiv.org/pdf/2306.05499 · arXiv
- Yi Liu Gelei Deng Zhengzi Xu Yuekang Li Yaowen Zheng Ying Zhang Lida Zhao Tianwei Zhang and Yang Liu. 2023. Jailbreaking chatgpt via prompt engineering: An empirical study. Retrieved from https://arxiv.org/pdf/2305.13860 · arXiv
- Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In Proceedings of the USENIX Security.
- Yang Liu Yuanshun Yao Jean-Francois Ton Xiaoying Zhang Ruocheng Guo Hao Cheng Yegor Klochkov Muhammad Faaiz Taufiq and Hang Li. 2023. Trustworthy LLMs: A survey and guideline for evaluating large language models’ alignment. Retrieved from https://arxiv.org/pdf/2308.05374 · arXiv
- Qinghua Lu Liming Zhu Xiwei Xu Zhenchang Xing Stefan Harrer and Jon Whittle. 2023. Building the future of responsible AI: A reference architecture for designing large language model based agents. Retrieved from https://arxiv.org/pdf/2311.13148 · arXiv
- Nikhil Mehta, Milagro Teruel, Xin Deng, Sergio Figueroa Sanz, Ahmed Awadallah, and Julia Kiseleva. 2024. Improving grounded language understanding in a collaborative environment by interacting with agents through help feedback. In Proceedings of the EACL.
- Kai Mei Zelong Li Shuyuan Xu Ruosong Ye Yingqiang Ge and Yongfeng Zhang. 2024. AIOS: LLM agent operating system. Retrieved from https://arxiv.org/pdf/2403.16971 · arXiv
- Lingbo Mo Zeyi Liao Boyuan Zheng Yu Su Chaowei Xiao and Huan Sun. 2024. A trembling house of cards? Mapping adversarial attacks against language agents. Retrieved from https://arxiv.org/pdf/2402.10196 · arXiv
- Sumeet Ramesh Motwani, Mikhail Baranchuk, Lewis Hammond, and Christian Schroeder de Witt. 2023. A perfect collusion benchmark: How can AI agents be prevented from colluding with information-theoretic undetectability?. In Proceedings of the NeurIPS Workshop.
- Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. 2024. Generating benchmarks for factuality evaluation of language models. In Proceedings of the EACL.
- Tai Nguyen and Eric Wong. 2023. In-context example selection with influences. Retrieved from https://arxiv.org/pdf/2302.11042 · arXiv
- Aidan O’Gara. 2023. Hoodwinked: Deception and cooperation in a text-based game for language models. Retrieved from https://arxiv.org/pdf/2308.01404 · arXiv
- Yikang Pan Liangming Pan Wenhu Chen Preslav Nakov Min-Yen Kan and William Yang Wang. 2023. On the risk of misinformation pollution with large language models. Retrieved from https://arxiv.org/pdf/2305.13661 · arXiv
- Rodrigo Pedro Daniel Castro Paulo Carreira and Nuno Santos. 2023. From prompt injections to SQL injection attacks: How protected is your LLM-integrated web application? Retrieved from https://arxiv.org/pdf/2308.01990 · arXiv
- Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. Retrieved from https://arXiv:2211.09527
- Steve Phelps and Rebecca Ranson. 2023. Of Models and Tin Men–a behavioural economics study of principal-agent problems in AI alignment using large-language models. Retrieved from https://arxiv.org/pdf/2307.11137 · arXiv
- Lukas Pöhler Valentin Schrader Alexander Ladwein and Florian von Keller. 2024. A technological perspective on misuse of available AI. Retrieved from https://arxiv.org/pdf/2403.15325 · arXiv
- Jack W. Rae Sebastian Borgeaud Trevor Cai Katie Millican Jordan Hoffmann Francis Song John Aslanides Sarah Henderson Roman Ring Susannah Young et al. 2021. Scaling language models: Methods analysis & insights from training Gopher. Retrieved from https://arxiv.org/pdf/2112.11446 · arXiv
- Fathima Abdul Rahman and Guang Lu. 2023. A contextualized real-time multimodal emotion recognition for conversational agents using graph convolutional networks in reinforcement learning. Retrieved from https://arxiv.org/pdf/2310.18363 · arXiv
- Alexander Robey Zachary Ravichandran Vijay Kumar Hamed Hassani and George J Pappas. 2024. Jailbreaking LLM-controlled robots. Retrieved from https://arxiv.org/pdf/2410.13691 · arXiv
- Sippo Rossi Alisia Marianne Michel Raghava Rao Mukkamala and Jason Bennett Thatcher. 2024. An early categorization of prompt injection attacks on large language models. Retrieved from https://arxiv.org/pdf/2402.00898 · arXiv
- Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the risks of LM agents with an LM-emulated sandbox. In Proceedings of the ICLR.
- Ahmed Salem Andrew Paverd and Boris Köpf. 2023. Maatphor: Automated variant analysis for prompt injection attacks. Retrieved from https://arxiv.org/pdf/2312.11513 · arXiv
- Leo Schwinn David Dobre Stephan Günnemann and Gauthier Gidel. 2023. Adversarial attacks and defenses in large language models: Old and new threats. Retrieved from https://arxiv.org/pdf/2310.19737 · arXiv
- Mrinank Sharma Meg Tong Tomasz Korbak David Duvenaud Amanda Askell Samuel R. Bowman Newton Cheng Esin Durmus Zac Hatfield-Dodds Scott R. Johnston et al. 2023. Towards understanding sycophancy in language models. Retrieved from https://arxiv.org/pdf/2310.13548 · arXiv
- Xinyue Shen Yixin Wu Michael Backes and Yang Zhang. 2024. Voice jailbreak attacks against GPT-4o. Retrieved from https://arxiv.org/abs/2405.19103 · arXiv
- Emily H. Soice Rafael Rocha Kimberlee Cordova Michael Specter and Kevin M. Esvelt. 2023. Can large language models democratize access to dual-use biotechnology? Retrieved from https://arxiv.org/pdf/2306.03809 · arXiv
- Yuxiang Sun Checheng Yu Junjie Zhao Wei Wang and Xianzhong Zhou. 2023. Self generated wargame AI: Double layer agent task planning based on large language model. Retrieved from https://arxiv.org/pdf/2312.01090 · arXiv
- Harini Suresh and John V. Guttag. 2019. A framework for understanding unintended consequences of machine learning. Retrieved from https://arxiv.org/pdf/1901.10002 · arXiv
- Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. 2024. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In Proceedings of the ICLR.
- Xiangru Tang Qiao Jin Kunlun Zhu Tongxin Yuan Yichi Zhang Wangchunshu Zhou Meng Qu Yilun Zhao Jian Tang Zhuosheng Zhang et al. 2024. Prioritizing safeguarding over autonomy: Risks of LLM agents for science. Retrieved from https://arxiv.org/pdf/2402.04247 · arXiv
- Mikhail Terekhov, Romain Graux, Eduardo Neville, Denis Rosset, and Gabin Kolly. 2023. Second-order Jailbreaks: Generative agents successfully manipulate through an intermediary. In Proceedings of the Multi-Agent Security Workshop @ NeurIPS’23. Retrieved from https://openreview.net/forum?id=HPmhaOTseN
- Yu Tian Xiao Yang Jingyuan Zhang Yinpeng Dong and Hang Su. 2023. Evil geniuses: Delving into the safety of llm-based agents. Retrieved from https://arxiv.org/pdf/2311.11855 · arXiv
- Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. 2024. Tensor trust: Interpretable prompt injection attacks from an online game. In Proceedings of the ICLR.
- Celine Wald and Lukas Pfahler. 2023. Exposing bias in online communities through large-scale language models. Retrieved from https://arxiv.org/pdf/2306.02294 · arXiv
- Eric Wallace Kai Xiao Reimar Leike Lilian Weng Johannes Heidecke and Alex Beutel. 2024. The instruction hierarchy: Training LLMs to prioritize privileged instructions. Retrieved from https://arxiv.org/pdf/2404.13208 · arXiv
- Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In Proceedings of the ICML.
- Fengxiang Wang Ranjie Duan Peng Xiao Xiaojun Jia YueFeng Chen Chongwen Wang Jialing Tao Hang Su Jun Zhu and Hui Xue. 2024. MRJ-Agent: An effective jailbreak agent for multi-round dialogue. Retrieved from https://arxiv.org/pdf/2411.03814 · arXiv
- Jerry Wei Chengrun Yang Xinying Song Yifeng Lu Nathan Hu Dustin Tran Daiyi Peng Ruibo Liu Da Huang Cosmo Du et al. 2024. Long-form factuality in large language models. Retrieved from https://arxiv.org/pdf/2403.18802 · arXiv
- Zeming Wei Yifei Wang and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. Retrieved from https://arxiv.org/pdf/2310.06387 · arXiv
- Roy Weiss Daniel Ayzenshteyn Guy Amit and Yisroel Mirsky. 2024. What was your prompt? A remote keylogging attack on AI assistants. Retrieved from https://arxiv.org/pdf/2403.09751 · arXiv
- Yotam Wolf Noam Wies Yoav Levine and Amnon Shashua. 2023. Fundamental limitations of alignment in large language models. Retrieved from https://arxiv.org/pdf/2304.11082 · arXiv
- Chen Henry Wu Jing Yu Koh Ruslan Salakhutdinov Daniel Fried and Aditi Raghunathan. 2024. Adversarial attacks on multimodal agents. Retrieved from https://arxiv.org/pdf/2406.12814v1 · arXiv
- Fangzhou Wu Shutong Wu Yulong Cao and Chaowei Xiao. 2024. WIPI: A new web threat for LLM-driven web agents. Retrieved from https://arxiv.org/pdf/2402.16965 · arXiv
- Junlin Xie Zhihong Chen Ruifei Zhang Xiang Wan and Guanbin Li. 2024. Large multimodal agents: A survey. Retrieved from https://arxiv.org/pdf/2402.15116 · arXiv
- Frank Xing. 2024. Designing heterogeneous LLM agents for financial sentiment analysis. Retrieved from https://arxiv.org/pdf/2401.05799 · arXiv
- Binfeng Xu Zhiyuan Peng Bowen Lei Subhabrata Mukherjee Yuchen Liu and Dongkuan Xu. 2023. Rewoo: Decoupling reasoning from observations for efficient augmented language models. Retrieved from https://arxiv.org/pdf/2305.18323 · arXiv
- Yuzhuang Xu Shuo Wang Peng Li Fuwen Luo Xiaolong Wang Weidong Liu and Yang Liu. 2023. Exploring large language models for communication games: An empirical study on werewolf. Retrieved from https://arxiv.org/pdf/2309.04658 · arXiv
- Zelai Xu Chao Yu Fei Fang Yu Wang and Yi Wu. 2023. Language agents with reinforcement learning for strategic play in the werewolf game. Retrieved from https://arxiv.org/pdf/2309.04658 · arXiv
- Xue Yan Yan Song Xinyu Cui Filippos Christianos Haifeng Zhang David Henry Mguni and Jun Wang. 2023. Ask more know better: Reinforce-learned prompt questions for decision making with large language models. Retrieved from https://arxiv.org/pdf/2310.18127 · arXiv
- Yong Yang Xuhong Zhang Yi Jiang Xi Chen Haoyu Wang Shouling Ji and Zonghui Wang. 2024. PRSA: Prompt reverse stealing attacks against large language models. Retrieved from https://arxiv.org/pdf/2402.19200 · arXiv
- Jingwei Yi Yueqi Xie Bin Zhu Keegan Hines Emre Kiciman Guangzhong Sun Xing Xie and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. Retrieved from https://arxiv.org/pdf/2312.14197 · arXiv
- Jiahao Yu Xingwei Lin and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. Retrieved from https://arxiv.org/pdf/2309.10253 · arXiv
- Zhiyuan Yu Xiaogeng Liu Shunning Liang Zach Cameron Chaowei Xiao and Ning Zhang. 2024. Don’t listen to me: Understanding and exploring jailbreak prompts of large language models. Retrieved from https://arxiv.org/pdf/2403.17336 · arXiv
- Yifan Zeng Yiran Wu Xiao Zhang Huazheng Wang and Qingyun Wu. 2024. AutoDefense: Multi-agent LLM defense against jailbreak attacks. Retrieved from https://arxiv.org/pdf/2403.04783 · arXiv
- Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua Tenenbaum, Tianmin Shu, and Chuang Gan. 2023. Building cooperative embodied agents modularly with large language models. In Proceedings of the NeurIPS Workshop.
- Wanpeng Zhang and Zongqing Lu. 2023. Rladapter: Bridging large language models to reinforcement learning in open worlds. Retrieved from https://arxiv.org/pdf/2309.17176 · arXiv
- Yuqi Zhang Liang Ding Lefei Zhang and Dacheng Tao. 2024. Intention analysis prompting makes large language models a good jailbreak defender. Retrieved from https://arxiv.org/pdf/2401.06561 · arXiv
- Zhexin Zhang Junxiao Yang Pei Ke and Minlie Huang. 2023. Defending large language models against jailbreaking attacks through goal prioritization. Retrieved from https://arxiv.org/pdf/2311.09096 · arXiv
- Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2024. Analyzing and mitigating object hallucination in large vision-language models. In Proceedings of the ICLR.
- Wei Zou Runpeng Geng Binghui Wang and Jinyuan Jia. 2024. PoisonedRAG: Knowledge poisoning attacks to retrieval-augmented generation of large language models. Retrieved from https://arxiv.org/pdf/2402.07867 · arXiv
Cited by
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“For instance, while pre-modern expert systems required manual updates to static knowledge bases, modern agents leverage emergent neural architectures to generalize across tasks [25].”
InfluentialAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
“For instance, while pre-modern expert systems required manual updates to static knowledge bases, modern agents leverage emergent neural architectures to generalize across tasks [25].”
InfluentialAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Exploring DeepSeek: A Survey on Advances, Applications, Challenges and Future Directions
Deng, Z., Ma, W., Han, Q., Zhou, W., Zhu, X., Wen, S., & Xiang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125498 · Cited by 224
Self-citationModel Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions
Hou, X., Zhao, Y., Wang, S., & Wang, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796519 · Cited by 167
Metadata
- Authors
- Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y.
- Affiliations
- Swinburne University of Technology (AU)
- Tsinghua University (CN)
- Ant Group (China) (CN)
- Venue
- ACM Computing Surveys
- Published
- 2025-02-21
- Volume
- 57
- Issue
- 7
- Pages
- 1-36
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4407236106
- Cited by
- 210
- References
- 92
- Citations by year
- 2026: 127 · 2025: 82
- FWCI
- 83.83
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Computer Networks and Communications › Access Control and Trust
- Keywords
- AI agents, AI agent security, security threats, autonomous software agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.