STAC: When Innocent Tools Form Dangerous Chains for LLM Agents
Open accessAbstract
As LLMs advance into autonomous agents with tool-use capabilities, they introduce security challenges that extend beyond traditional content-based LLM safety concerns. This paper introduces Sequential Tool Attack Chaining (\STAC), a novel multi-turn attack framework that exploits agent tool use. \STAC chains together tool calls that each appear harmless in isolation but, when combined, collectively enable harmful operations that only become apparent at the final execution step. At the core of \STAC is an automated, closed-loop pipeline that synthesizes executable multi-step tool chains, validates them through in-environment execution, and reverse-engineers stealthy multi-turn prompts that reliably induce agents to execute the verified malicious sequence. Using this framework, we generate and systematically evaluate 483 \STAC cases, featuring 1,352 sets of user-agent-environment interactions and spanning diverse domains, tasks, agent types, and 10 failure modes. Our evaluations show that state-of-the-art LLM agents are highly vulnerable to \STAC, with an average final attack success rate (ASR) of 91.2\% -- exceeding 90\% for all but one of the eight agents evaluated. We further perform defense analysis and find that existing prompt-based defenses provide limited protection. To address this gap, we propose a new reasoning-driven defense prompt that achieves the strongest initial-turn protection, cutting ASR by up to 28.8\%; however, this advantage erodes sharply under adaptive attacks, and an experience-based defense (ToolShield) proves more durable over sustained multi-turn interactions. These results highlight a crucial gap: defending tool-enabled agents requires reasoning over entire action sequences and their cumulative effects, rather than evaluating isolated prompts or responses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
STAC demonstrates tool-chain attacks with greater than 90% ASR on GPT-4.1.
“STAC [12] demonstrates tool-chain attacks with > 90% ASR on GPT-4.1.”
From SentinelAgent: Intent-Verified Delegation Chains for Securing Federal Multi-Agent AI Systems (2026) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
Self-citationAgent-SafetyBench: Evaluating the Safety of LLM Agents
Zhang, Z., Cui, S., Lu, Y., Zhou, J., Yang, J., Wang, H., & Huang, M. (2024) · arXiv · doi:10.48550/arxiv.2412.14470 · Cited by 6 · see citing works
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
Vijayvargiya, S., Soni, A. B., Zhou, X., Wang, Z. Z., Dziri, N., Neubig, G., & Sap, M. (2025) · arXiv · doi:10.48550/arxiv.2507.06134 · cited 1×
“…assess whether STAC transfers beyond simulated environments, we apply it to real MCP (Model Con-text Protocol) tool backends (e.g., filesystem and browser automation servers running in Docker containers) using the OpenAgentSafety (OAS) benchmark (Vijayvargiya et al., 2025) (details in Appendix G).”
Flagged first, then by citations
Research works printed but not linked: 53
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
- Abdul Ahad, Md Hossain, Mahbubul Alam, Sai Puppala, Jinsun Lee, Md Tanvir Alam, and Ashaduzzaman Talukder. 2026. Semantic intent fragmentation: A single-shot compositional attack on multi-agent AI pipelines. arXiv preprint arXiv:2604.08608. · arXiv
- Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. 2024. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024. · arXiv
- Rishika Bhagwatkar, Kevin Kasa, Abhay Puri, Gabriel Huang, Irina Rish, Graham W. Taylor, Krishna- murthy Dj Dvijotham, and Alexandre Lacoste. 2025. Indirect prompt injections: Are firewalls all you need, or stronger benchmarks? arXiv preprint arXiv:2510.05244. · arXiv
- Si Chen, Xiao Yu, Ninareh Mehrabi, Rahul Gupta, Zhou Yu, and Ruoxi Jia. 2025. Strategize globally, adapt locally: A multi-turn red teaming agent with dual- level learning. arXiv preprint arXiv:2504.01278. · arXiv
- Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramèr. 2025. Defeating prompt injections by design. arXiv preprint arXiv:2503.18813. · arXiv
- Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents. arXiv e-prints, pages arXiv-2406.
- Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. 2024. Redcode: Risky code execution and generation benchmark for code agents. Advances in Neural In- formation Processing Systems, 37:106190-106236.
- Weiyang Guo, Jing Li, Wenya Wang, Yu Li, Daojing He, Jun Yu, and Min Zhang. 2025. Mtsa: Multi- turn safety alignment for llms through multi-round red-teaming. arXiv preprint arXiv:2505.17147. · arXiv
- Wenbo Guo, Wei Zeng, Chengwei Liu, Xiaojun Jia, Yijia Xu, Lei Tang, Yong Fang, and Yang Liu. 2026. MalSkillBench: A runtime-verified bench- mark of malicious agent skills. arXiv preprint arXiv:2606.07131. · arXiv
- Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. 2024. Defending against indirect prompt injection attacks with spotlighting. arXiv preprint arXiv:2403.14720. · arXiv
- Hanjiang Hu, Alexander Robey, and Changliu Liu. 2025. Steering dialogue dynamics for robustness against multi-turn jailbreaking attacks. arXiv preprint arXiv:2503.00187. · arXiv
- Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gi- anna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mis- tral 7b. ArXiv, abs/2310.06825. · arXiv
- Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun van der Weij, Linda Petrini, Chen Bo Calvin Zhang, John Hughes, Xiang Deng, Henry Sleight, Tyler Tracy, et al. 2025. Shade-arena: Evaluating sabo- tage and monitoring in llm agents. arXiv preprint arXiv:2506.15740. · arXiv
- Jing-Jing Li, Valentina Pyatkin, Max Kleiman- Weiner, Liwei Jiang, Nouha Dziri, Anne Collins, Jana Schaich Borg, Maarten Sap, Yejin Choi, and Sydney Levine. 2025. Safetyanalyst: Interpretable, transparent, and steerable safety moderation for ai behavior. In Forty-second International Conference on Machine Learning.
- Xu Li, Simon Yu, Minzhou Pan, Yiyou Sun, Bo Li, Dawn Song, Xue Lin, and Weiyan Shi. 2026a. Un- safer in many turns: Benchmarking and defending multi-turn safety risks in tool-using agents. arXiv preprint arXiv:2602.13379. · arXiv
- Yuxuan Li, Ming Li, Zhe Ma, Fan Zhu, Ding Liu, Wei Wang, and Fuli Feng. 2026b. Plant, persist, trigger: Sleeper attack on large language model agents. arXiv preprint arXiv:2605.28201. · arXiv
- Xiaofeng Lin, Yukai Yang, Daniel Guo, Sahil Arun Nale, Charles Fleming, and Guang Cheng. 2026. Context-fractured decomposition attacks on tool- using LLM agents: Exploiting artifact provenance gaps. arXiv preprint arXiv:2606.09084. · arXiv
- Kanghua Mo, Li Hu, Yucheng Long, and Zhihao Li. 2025. Attractive metadata attack: Inducing llm agents to invoke malicious tools. arXiv preprint arXiv:2508.02110. · arXiv
- Kyle O'Brien, David Majercak, Xavier Fernandes, Richard Edgar, Blake Bullwinkel, Jingya Chen, Har- sha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangdeh. 2024. Steering language model refusal with sparse autoencoders. arXiv preprint arXiv:2411.11296. · arXiv
- Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. · arXiv
- Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel. 2025. X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents. arXiv preprint arXiv:2504.13203. · arXiv
- Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Bar- mentlo, Karmesh Yadav, Kartik Khandelwal, Khy- athi Raghavi Chandu, et al. 2025. Magistral. arXiv preprint arXiv:2506.10910. · arXiv
- Yangjun Ruan, Honghua Dong, Andrew Wang, Sil- viu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2023. Identifying the risks of lm agents with an lm- emulated sandbox. arXiv preprint arXiv:2309.15817. · arXiv
- Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2024. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack. arXiv preprint arXiv:2404.01833, 2(6):17. · arXiv
- Chris Schneider, Kriti Faujdar, Philipp Schoenegger, and Ben Bariach. 2026. Securing multi-tool AI agent chains with dynamic, real-time compositional poli- cies. arXiv preprint arXiv:2607.03423. · arXiv
- Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, and Diyi Yang. 2024. Privacylens: Evaluating privacy norm awareness of language models in action. Ad- vances in Neural Information Processing Systems, 37:89373-89407.
- Jonathan Steinberg and Oren Gal. 2026. MOSAIC- Bench: Measuring compositional vulnerability induction in coding agents. arXiv preprint arXiv:2605.03952. · arXiv
- Harold Triedman, Rishi Jha, and Vitaly Shmatikov. 2025. Multi-agent systems execute arbitrary mali- cious code. arXiv preprint arXiv:2503.12188. · arXiv
- Ada Defne Tur, Nicholas Meade, Xing Han Lù, Ale- jandra Zambrano, Arkil Patel, Esin Durmus, Span- dana Gella, Karolina Stańczak, and Siva Reddy. 2025. Safearena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957. · arXiv
- Lichao Wang, Zhihao Ren, Tao Yang, Jiaming Ji, Chi Harold Liu, Yaodong Yang, and Juntao Dai. 2026a. SafeMCP: Proactive power regulation for LLM agent defense via environment-grounded look- ahead reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Lin- guistics (ACL).
- Su Wang, Pin Qian, Yihang Chen, Junxian You, Xi- aoyuan Wang, Xiaochong Jiang, Lifei Liu, Haoran Yu, and Jingzhou Xu. 2026b. When safe skills col- lide: Measuring compositional risk in agent skill ecosystems. arXiv preprint arXiv:2606.00448. · arXiv
- Xunguang Wang, Daoyuan Wu, Zhenlan Ji, Zongjie Li, Pingchuan Ma, Shuai Wang, Yingjiu Li, Yang Liu, Ning Liu, and Juergen Rahmel. 2024. Selfdefend: Llms can defend themselves against jailbreaking in a practical manner. arXiv preprint arXiv:2406.05498. · arXiv
- Yuhui Wang, Tanqiu Jiang, Jiacheng Liang, Charles Fleming, and Ting Wang. 2026c. MAGE: Safeguard- ing LLM agents against long-horizon threats via shadow memory. arXiv preprint arXiv:2605.03228. · arXiv
- Shihao Weng, Yang Feng, Jinrui Zhang, Xiaofei Xie, Jiongchi Yu, and Jia Liu. 2026. ARGUS: Defending LLM agents against context-aware prompt injection. arXiv preprint arXiv:2605.03378. · arXiv
- Benlong Wu, Weiming Zhang, Kejiang Chen, Han Fang, and Nenghai Yu. 2026a. Provably secure agent guardrail. arXiv preprint arXiv:2605.29251. · arXiv
- Jiangrong Wu, Huaijin Wang, Yihao Zhang, Yuhong Nan, and Shuai Wang. 2026b. MOSAIC: Knowledge- guided CLI command composition attack in LLM coding agents. arXiv preprint arXiv:2607.02857. · arXiv
- Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. 2024. Os-copilot: Towards gener- alist computer agents with self-improvement. arXiv preprint arXiv:2402.07456. · arXiv
- Yi Xie, Jiawei Du, Yu Cheng, Jiuan Zhou, and Zhaoxia Yin. 2026. Benign in isolation, harmful in composi- tion: Security risks in agent skill ecosystems. arXiv preprint arXiv:2606.15242. · arXiv
- Siqiao Xue, Caigao Jiang, Wenhui Shi, Fangyin Cheng, Keting Chen, Hongjun Yang, Zhiping Zhang, Jian- shan He, Hongyang Zhang, Ganglin Wei, et al. 2023. Db-gpt: Empowering database interactions with private large language models. arXiv preprint arXiv:2312.17449. · arXiv
- An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388. · arXiv
- Jirui Yang, Zheyu Lin, Shuhan Yang, Zhihui Lu, and Xin Du. 2025b. Concept enhancement engineering: A lightweight and efficient robust defense against jailbreak attacks in embodied ai. arXiv preprint arXiv:2504.13201. · arXiv
- John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024a. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528- 50652.
- Xikang Yang, Xuehai Tang, Songlin Hu, and Jizhong Han. 2024b. Chain of attack: a semantic-driven con- textual multi-turn attacker for llm. arXiv preprint arXiv:2405.05610. · arXiv
- Zonghao Ying, Deyue Zhang, Zonglei Jing, Yisong Xiao, Quanchen Zou, Aishan Liu, Siyuan Liang, Xiangzheng Zhang, Xianglong Liu, and Dacheng Tao. 2025. Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models. arXiv preprint arXiv:2502.11054. · arXiv
- Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. 2025. Optimizing generative ai by backpropagating language model feedback. Nature, 639(8055):609-616.
- Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persua- sion to challenge ai safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 14322-14350.
- Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. 2025. Adaptive attacks break de- fenses against indirect prompt injection attacks on llm agents. arXiv preprint arXiv:2503.00061. · arXiv
- Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691. · arXiv
- Yi Zhao and Youzhi Zhang. 2025. Siren: A learning- based multi-turn attack framework for simulating real-world human jailbreak behaviors. arXiv preprint arXiv:2501.14250. · arXiv
- Kaiwen Zhou, Xuandong Zhao, Gaowen Liu, Jayanth Srinivasa, Aosong Feng, Dawn Song, and Xin Eric Wang. 2025. Safekey: Amplifying aha-moment insights for safety reasoning. arXiv preprint arXiv:2505.16186. · arXiv
- Xuhui Zhou, Hyunwoo Kim, Faeze Brahman, Liwei Jiang, Hao Zhu, Ximing Lu, Frank Xu, Bill Yuchen Lin, Yejin Choi, Niloofar Mireshghallah, et al. 2024. Haicosystem: An ecosystem for sandboxing safety risks in human-ai interactions. arXiv preprint arXiv:2409.16427. · arXiv
- Pengyu Zhu, Zhenhong Zhou, Yuanhe Zhang, Shilinlu Yan, Kun Wang, and Sen Su. 2025. Demona- gent: Dynamically encrypted multi-backdoor im- plantation attack on llm-based agent. arXiv preprint arXiv:2502.12575. · arXiv
Cited by
Sorted by citations
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 2 · see citing works
- Authors
- Li, J.-J., He, J., Shang, C., Kulshreshtha, D., Xian, X., Zhang, Y., Su, H., Swamy, S., & Qi, Y.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Pages
- 226-262
- arXiv
- 2509.25624
- OpenAlex
- W7084069946
- Cited by
- 0
- References
- 3
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Workshop for Research on Agent Language Models (REALM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Keywords
- LLM agents, tool use, LLM security, prompt engineering, attack success rate, autonomous agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
