StruQ: defending against prompt injection with structured queries
Open accessAbstract
Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model into deviating from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that supports structured queries. This system is made of (1) a secure front-end that formats a prompt and user data into a special format, and (2) a specially trained LLM that can produce high-quality outputs from these inputs. The LLM is trained using a novel fine-tuning strategy: we convert a base (non-instruction-tuned) LLM to a structured instruction-tuned model that will only follow instructions in the prompt portion of a query. To do so, we augment standard instruction tuning datasets with examples that also include instructions in the data portion of the query, and fine-tune the model to ignore these. Our system significantly improves resistance to prompt injection attacks, with little or no impact on utility. Our code is released at https://github.com/Sizhe-Chen/StruQ.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our work differs from Chen et al. (2024) in that we focus on a hierarchy of instructions with multiple levels, whereas they focus specifically on system messages versus user messages.
“Our work differs in that we focus on a hierarchy of instructions with multiple levels, whereas they focus specifically on system messages versus user messages.”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Chen et al., 2024
The Prompt Injection Defenses section says StruQ has two types of messages—prompt and data—not system and user messages. Section 4.2 likewise describes training the model to follow instructions in the prompt but not the data. The paper treats system prompts as a possible future extension, so the claimed focus is contrary to its description of StruQ. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredRisks driven by write capability increase when T_execute is non-empty, while strong, non-bypassable gates reduce the likelihood that these risks propagate into executed actions.
“Risks driven by write capability increase when T execute k ̸ = ∅, while strong, non-bypassable gates reduce the likelihood that these risks propagate into executed actions [1], [3], [100], [101], [99].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [100]
Section 4.2 describes a front-end that filters reserved delimiters so attackers cannot spoof the boundary between instructions and data, and the B.2 results show lower prompt-injection success rates for many attacks. The paper does not establish the claim about non-empty T_execute, write capability, or gates preventing risks from becoming executed actions; it evaluates LLM responses rather than downstream action execution. (LLM, conf=0.81)
Experimental — accuracy not yet measuredChen et al. (2024) proposed to train LLMs to ignore instructions provided in the user input.
“In particular, Chen et al. (2024) proposed to train LLMs to ignore instructions provided in the user input.”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Chen et al., 2024
Section 4.4 (Structured Instruction Tuning) says the model is trained to ignore instructions in the data portion of a query while obeying instructions in its prompt portion. The claim omits that distinction: the paper does not propose ignoring all instructions provided in user input. (LLM, conf=0.84)
Experimental — accuracy not yet measuredNone of the heuristic prompt injection defenses provide any guarantee of security and regularly fall short to new attacks in practice.
“Unfortunately, none of these heuristic defenses provide any guarantee of security and regularly fall short to new attacks in practice.”
From Defeating Prompt Injections by Design (2026) · cites this paper as Chen et al., 2024
Chen et al. (2024) train models to completely ignore all instructions in the user messages, whereas we train models to conditionally follow lower-level instructions when applicable.
“Moreover, they train models to completely ignore all instructions in the user messages, whereas we train models to conditionally follow lower-level instructions when applicable.”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Chen et al., 2024
Recent work has advocated for teaching a model to treat third-party user inputs as data, not as instructions.
“For prompt injection on closed-domain tasks (Section 3.2), recent work has advocated for teaching a model to treat third-party user inputs as data, not as instructions (Chen et al., 2024;Willison, 2023;Zverev et al., 2024;Yi et al., 2023;Liu et al., 2023).”
From The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024) · cites this paper as Chen et al., 2024
The paper does not evaluate partially-effective defensive approaches proposed in the literature.
“Thus while a few partially-effective defensive approaches have been proposed in the literature, we do not evaluate them here (Yi et al., 2023;Piet et al., 2023;Chen et al., 2024;Wallace et al., 2024).”
From Ranking Manipulation for Conversational Search Engines (2024) · cites this paper as Chen et al., 2024
The gate returns 1 only when the action is permitted under policy Π, supported by the required evidence, satisfies the relevant invariants I, and passes the gate conditions in G k.
“which returns 1 only when the action is permitted under policy Π, supported by the required evidence, satisfies the relevant invariants I, and passes the gate conditions in G k [3], [1], [100], [101], [69].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [100]
The secure front-end discussion (section 5.4 and the filtering description; page unavailable) describes separating trusted instructions from untrusted data and filtering delimiter tokens. It does not describe a gate that returns 1 only after checking policy Π, required evidence, invariants I, and conditions G k. Searches for those gate and authorization terms found no passage addressing that predicate. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThe scope of coverage and the risks decide if an agent can carry out a plan, not just if they can have a major effect.
“The scope of coverage and the risks decide if an agent can carry out a plan, not just if they can have a major effect [1], [3], [100], [119], [31].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [100]
The Problem Statement (§3.1) defines prompt injection, and the Discussion (§6) describes the limits of StruQ's protection. Neither addresses whether coverage and risks determine an agent's ability to carry out a plan, as distinct from its potential impact. Searches for that wording and for agent plans, coverage, and risk found no supporting passage in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 43
- Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023. · arXiv
- Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injec- tion. arXiv:2302.12173, 2023. · arXiv
- Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmark- ing prompt injection attacks and defenses. In USENIX Security Symposium, 2024.
- Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game. In International Conference on Learning Representations (ICLR), 2024.
- Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop, 2022.
- Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. Jatmo: Prompt injection defense by task-specific finetuning. In European Symposium on Research in Computer Security (ESORICS), 2023.
- Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail? In Neural Information Processing Systems, 2023.
- Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jail- breaking Black Box Large Language Models in Twenty Queries, 2023. arXiv:2310.08419. · arXiv
- Kaijie Zhu et al. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. arXiv:2306.04528, 2023. · arXiv
- Jindong Wang et al. On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective. ICLR 2023 Workshop on Trustworthy and Reliable Large- Scale Machine Learning Models, 2023.
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971, 2023. · arXiv
- Albert Q. Jiang et al. Mistral 7B, 2023. arXiv:2310.06825. · arXiv
- Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box LLMs automatically. arXiv:2312.02119, 2023. · arXiv
- Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrik- son. Universal and transferable adversarial attacks on aligned language models. arXiv:2307.15043, 2023. · arXiv
- Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero- shot learners. In International Conference on Learning Representations, 2021.
- Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tian- wei Zhang, Fei Wu, and Guoyin Wang. Instruc- tion Tuning for Large Language Models: A Survey. arXiv:2308.10792, 2023. · arXiv
- Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforce- ment learning from human feedback. arXiv:2204.05862, 2022. · arXiv
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Pro- cessing Systems (NeurIPS), pages 27730-27744, 2022.
- Jean Kaddour, Joshua Harris, Maximilian Mozes, Her- bie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv:2307.10169, 2023. · arXiv
- Hezekiah J Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. Evaluat- ing the susceptibility of pre-trained language models via handcrafted adversarial examples. arXiv:2209.02128, 2022. · arXiv
- Jiahao Yu, Yuhang Wu, Dong Shu, Mingyu Jin, and Xinyu Xing. Assessing Prompt Injection Risks in 200+ Custom GPTs. arXiv:2311.11538, 2023. · arXiv
- Daniel Wankit Yip, Aysan Esmradi, and Chun Fai Chan. A novel evaluation framework for assessing resilience against prompt injection attacks in large language mod- els. In 2023 IEEE Asia-Pacific Conference on Com- puter Science and Data Engineering (CSDE), pages 1-5, 2023.
- Philipp Vogt, Florian Nentwich, Nenad Jovanovic, Engin Kirda, Christopher Kruegel, and Giovanni Vigna. Cross site scripting prevention with dynamic data tainting and static analysis. In NDSS, 2007.
- William G Halfond, Jeremy Viegas, Alessandro Orso, et al. A classification of SQL-injection attacks and coun- termeasures. In Proceedings of the IEEE international symposium on secure software engineering, 2006.
- Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv:2312.14197, 2023. · arXiv
- Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Jo- hannes Heidecke, and Alex Beutel. The Instruction Hierarchy: Training LLMs to Prioritize Privileged In- structions. arXiv:2404.13208, 2024. · arXiv
- Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis- François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan Boyd-Graber. Ignore this title and hack- aprompt: Exposing systemic vulnerabilities of llms through a global scale prompt hacking competition. arXiv:2311.16119, 2023. · arXiv
- Xuchen Suo. Signed-prompt: A new approach to prevent prompt injection attacks against llm-integrated applica- tions. arXiv:2401.07612, 2024. · arXiv
- Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How Robust is Google's Bard to Adversarial Image Attacks? arXiv:2309.11751, 2023. · arXiv
- Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in- context demonstrations. In International Conference on Machine Learning (ICML), 2024.
- Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. Tricking llms into disobedience: Understanding, analyzing, and preventing jailbreaks. arXiv:2305.14965, 2023. · arXiv
- Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. MasterKey: Automated jailbreak across mul- tiple large language model chatbots. arXiv:2307.08715, 2023. · arXiv
- Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "Do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv:2308.03825, 2023. · arXiv
- Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv:2310.04451, 2023. · arXiv
- Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, pages 2633-2650, 2021.
- Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. Bag of tricks for training data extraction from language mod- els. In International Conference on Machine Learning (ICML), pages 40306-40320, 2023.
- Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christo- pher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv:2311.17035, 2023. · arXiv
- Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. Ana- lyzing leakage of personally identifiable information in language models. In IEEE Symposium on Security and Privacy (SP), pages 346-363, 2023.
- Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jailbreak- ing privacy attacks on chatgpt. In The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
- Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Backdoor Attacks for In-Context Learning with Language Models. In ICML Workshop on Adversarial Machine Learning, 2023.
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
- Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
- David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Pa- pernot, and Vardan Papyan. Llm censorship: A ma- chine learning challenge or a computer security prob- lem? arXiv:2307.10719, 2023. · arXiv
Cited by
Sorted by citations
Friha, O., Ferrag, M. A., Kantarcı, B., Çakmak, B., Ozgun, A., & Ghoualmi‐Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 209 · see citing works
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.624 · Cited by 147 · see citing works
PromptShield: Deployable Detection for Prompt Injection Attacks
Jacob, D., Alzahrani, H., Hu, Z., Alomair, B., & Wagner, D. (2024) · ACM Conference on Data and Application Security and Privacy (CODASPY) · doi:10.1145/3714393.3726501 · Cited by 35 · see citing works
Self-citationAgentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28 · see citing works
DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks
Liu, Y., Jia, Y.-Q., Jia, J.-Y., Song, D., & Gong, N. (2025) · IEEE Symposium on Security and Privacy · doi:10.1109/sp61157.2025.00250 · Cited by 203 · see citing works
He, Y., Wang, E., Rong, Y., Cheng, Z., & Chen, H. (2025) · 2025 IEEE/ACM International Workshop on Responsible AI Engineering (RAIE) · doi:10.1109/raie66699.2025.00013 · Cited by 19 · see citing works
Defense against prompt injection attack by leveraging attack techniques
Chen, Y., Li, H., Zheng, Z., Wu, D., Song, Y., & Hooi, B. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.897 · Cited by 19 · see citing works
Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
Lee, D., & Tiwari, M. (2024) · arXiv · doi:10.48550/arxiv.2410.07283 · Cited by 18 · see citing works
Defeating Prompt Injections by Design
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F. (2026) · 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml68715.2026.00040 · Cited by 15 · see citing works
“In addition, some researchers explore training or fine-tuning models to become more resilient to prompt injections, enabling them to discern and ignore malicious instructions [17, 27, 28].”
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024) · arXiv · doi:10.48550/arxiv.2404.13208 · Cited by 14 · see citing works
“Defenses for Prompt Injection For prompt injection on closed-domain tasks (Section 3.2), recent work has advocated for teaching a model to treat third-party user inputs as data, not as instructions (Chen et al., 2024; Willison, 2023; Zverev et al., 2024; Yi et al., 2023; Liu et al., 2023).”
BackgroundInfluentialGet My Drift? Catching LLM Task Drift with Activation Deltas
Abdelnabi, S., Fay, A., Cherubin, G., Salem, A., Fritz, M., & Paverd, A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00011 · Cited by 20 · see citing works
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11 · see citing works
Ranking Manipulation for Conversational Search Engines
Pfrommer, S., Bai, Y., Gautam, T., & Sojoudi, S. (2024) · arXiv · doi:10.48550/arxiv.2406.03589 · Cited by 8 · see citing works
“Thus while a few partially-effective defensive approaches have been proposed in the literature, we do not evaluate them here (Yi et al., 2023; Piet et al., 2023; Chen et al., 2024; Wallace et al., 2024).”
BackgroundAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024) · arXiv · doi:10.48550/arxiv.2406.13352 · Cited by 7 · see citing works
“Most models incur a loss of 10%–25% in absolute utility under attack.”
BackgroundPrevention of Prompt Injection Attacks Over Financial Applications Integrated with LLM
Joshi, T., Naik, V., Mistry, I., & Mangrulkar, R. S. (2025) · International Conference on Advancement in Computation & Computer Technologies (InCACCT) · doi:10.1109/incacct65424.2025.11011372 · Cited by 7 · see citing works
Design Patterns for Securing LLM Agents against Prompt Injections
Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., Fischer, M., Froelicher, D., Grosse, K., Naeff, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.08837 · Cited by 2 · see citing works
“…various defensive measures against prompt injections, from heuristic approaches like detection and adversarial training (Wallace et al., 2024; Chen et al., 2024; Zverev et al., 2024; Willison, 2023a; Yi et al., 2023; Zou et al., 2024), to principled system-level isolation mechanisms…”
Progent: Securing AI Agents with Privilege Control
Shi, T., He, J., Wang, Z., Li, H., Wu, L., Guo, W., & Song, D. (2025) · arXiv · doi:10.48550/arxiv.2504.11703 · Cited by 3 · see citing works
Samonte, M. J. C., Aparize, J. E. R., Gonzales, E. J. S., & Morilla, J. L. (2024) · IEEE International Conference on Information, Communication and Networks (ICICN) · doi:10.1109/icicn62625.2024.10761619 · Cited by 2 · see citing works
Adversarial Search Engine Optimization for Large Language Models
Nestaas, F., Debenedetti, E., & Tramèr, F. (2024) · arXiv · doi:10.48550/arxiv.2406.18382 · Cited by 1 · see citing works
“Many of our attacks rely on prompt injections, which exploit LLMs inability to reliably distinguish between data and instructions [5, 9, 10, 15].”
BackgroundWASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
Evtimov, I., Zharmagambetov, A., Grattafiori, A., Guo, C., & Chaudhuri, K. (2025) · arXiv · doi:10.48550/arxiv.2504.18575 · Cited by 1 · see citing works
“More advanced methods involve altering the fine-tuning process to enhance defense performance (Piet et al., 2023; Chen et al., 2024a,b; Yi et al., 2023; Wallace et al., 2024; Wu et al., 2024c).”
EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
Liao, Z., Mo, L., Xu, C., Kang, M., Zhang, J., Xiao, C., Tian, Y., Li, B., & Sun, H. (2024) · arXiv · doi:10.48550/arxiv.2409.11295 · Cited by 1 · see citing works
“Although we have demonstrated that one specific type of system prompt defense cannot mitigate the EIA, we acknowledge that other works (Chen et al., 2024; Wallace et al., 2024) have proposed methods to prioritize instructions over data to counter injection attacks.”
AI safety: a climb to Armageddon?
Cappelen, H., Dever, J., & Hawthorne, J. T. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02297-w · Cited by 1 · see citing works
Dissecting Adversarial Robustness of Multimodal LM Agents
Wu, C. H., Shah, R., Koh, J. Y., Salakhutdinov, R., Fried, D., & Raghunathan, A. (2024) · arXiv · doi:10.48550/arxiv.2406.12814 · Cited by 1 · see citing works
CAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents
Hossain, S. M. A., Shayoni, R. K., & Morol, M. K. (2026) · arXiv · doi:10.48550/arxiv.2609.02265
“Injection defenses refuse content whose provenance is not trusted (Chen et al., 2025; Hines et al., 2024; Debenedetti et al., 2025), which is sound when the threat is external but blind to the case that matters here, where a legitimate user revises a legitimate belief through a legitimate channel.”
MAPLE-Guard: Memory-Aware Link Enforcement Against Memory-Link Poisoning in Multi-Agent Systems
Xiong, W., Zhou, Y., Wang, J., Gu, S., Tang, B., Li, Z., Xiong, F., Wen, Y., & Wen, M. (2026) · arXiv · doi:10.48550/arxiv.2608.00426
“StruQ combines a secure front end with instruction/data-separation training [2].”
SoK: The Attack Surface of Agentic AI - Tools and Autonomy
Dehghantanha, A., & Homayoun, S. (2026) · arXiv · doi:10.48550/arxiv.2603.22928
Multi-Agent Systems Execute Arbitrary Malicious Code
Triedman, H., Jha, R., & Shmatikov, V. (2025) · arXiv · doi:10.48550/arxiv.2503.12188
Roesner, F., & Kohno, T. (2026) · arXiv · doi:10.48550/arxiv.2609.17817
HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety
Bai, Y., Duan, J., Peng, J., Wu, X., Liu, S., Wang, S., & Chen, T. (2026) · arXiv · doi:10.48550/arxiv.2608.17597
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075
Probing Robustness of In-context Learning in LLM Classification Predictions Under Label Noise
Ayachitula, S., Kundu, C. K., & Mishra, B. K. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00057
Srour, A., Mor, A. G., Omar, M., Barash, Y., Litmanovich, D., Srour-Asmar, M., Klang, E., & Gorenshtein, A. (2026) · Research Square · doi:10.21203/rs.3.rs-10950882/v1
LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
Li, S., Xu, C., Wang, J., Gong, X., Chen, C., Zhang, J., Wang, J., Lam, K.-Y., & Ji, S. (2025) · arXiv · doi:10.48550/arxiv.2506.09443
“To account for this bias, evaluations are typically conducted using both input orders, with the results defined as:”
Prompt Injection Attack to Tool Selection in LLM Agents
Shi, J., Yuan, Z., Tie, G., Zhou, P., Gong, N. Z., & Sun, L. (2025) · arXiv · doi:10.48550/arxiv.2504.19793 · Cited by 2 · see citing works
- Authors
- Chen, S., Piet, J., Sitawarin, C., & Wagner, D.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.06363
- OpenAlex
- W4391766701
- Cited by
- 10
- References
- 0
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2229876, 2154873)
- U.S. Department of Homeland Security
- Open Philanthropy Project
- King Abdulaziz City for Science and Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- prompt injection, structured queries, large language models, instruction following, instruction tuning, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
