The DOI matches its Crossref record; 31 of the 41 references it cites were found in a registry or an index; 4 of the 17 sources its claims cite were checked for support.
Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents
Open accessAbstract
Large Language Model (LLM) agents exhibit remarkable performance across diverse applications by using external tools to interact with environments.However, integrating external tools introduces security risks, such as indirect prompt injection (IPI) attacks.Despite defenses designed for IPI attacks, their robustness remains questionable due to insufficient testing against adaptive attacks.In this paper, we evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%.This reveals critical vulnerabilities in current defenses.Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper evaluates eight defenses against indirect prompt injection (IPI) attacks on LLM agents and designs adaptive attacks that bypass all of them, with adaptive attack success rates consistently above 50%. It argues that adaptive attack evaluation is necessary to assess defense robustness.
Key findings
- Adaptive attacks broke all eight defenses across both tested agents, with ASR-adaptive attack exceeding 50% in all cases.
- For the Vicuna-7B prompted agent, most defenses reduced original ASR from 56% to lower values, such as 12% with adversarial finetuning; for the Llama3-8B finetuned agent, original ASR was 9% and lower under defenses.
- Detection-based defenses were largely bypassed: fine-tuned detector detection rate dropped from 61% on original attacks to 1% with adversarial strings for Vicuna-7B; LLM-based detector dropped from 72% to 0% for Llama3-8B; perplexity filtering had 0% detection on original attacks and 1% with adversarial strings.
- Adaptive strings also improved ASR over non-adaptive strings for defenses targeting adversarial strings: perplexity filtering Vicuna-7B 24% → 76%, Llama3-8B 22% → 61%; paraphrasing Vicuna-7B 53% → 79%, Llama3-8B 8% → 54%.
- Detailed ASRs: Vicuna-7B total original 0.50, ASR-defense 0.33 ± 0.18, ASR-attack 0.68 ± 0.15; Llama3-8B total original 0.40, ASR-defense 0.02 ± 0.02, ASR-attack 0.57 ± 0.17.
Methods
- Evaluated eight defenses grouped as detection-based (fine-tuned detector, LLM-based detector, perplexity filtering), input-level (instructional prevention, data prompt isolation, sandwich prevention, paraphrasing), and model-level (adversarial finetuning).
- Used InjecAgent benchmark subset of 100 test cases: 50 direct harm and 50 data stealing, including 25 unique direct harm attacks, 27 distinct data stealing attacks, and 16 user scenarios. Tested two agents: Vicuna-7B prompted ReAct agent and Llama3-8B finetuned agent.
- Designed adaptive attacks using GCG, multi-objective GCG, two-stage GCG, and AutoDAN, assuming white-box access to agent, defenses, and prompts. Metrics included attack success rate (ASR), target rate, and detection rate; additional AgentDojo experiments used 16 cases on Llama3-8B.
Limitations
- The paper states its adaptive attacks focus only on the agent’s first-step action, so they provide the least improvement in the second step of data-stealing attacks.
- The paper states attacks assume white-box access to the agent model, defense models, and detailed prompts, and that it does not account for combinations of defenses or exhaustively cover all possible defenses.
- Evident from the text: the main evaluation uses a 100-case subset of InjecAgent, and the AgentDojo extension uses only 16 cases, which may limit generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
22 Own result · 12 Own interpretation · 9 No basis given · 5 Not checked yet · 4 Sources not found · 3 Own method · 2 Supported · 1 Partly supported · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Another method is to paraphrase the external content E Tu to disrupt tokenlevel optimized adversarial strings (Jain et al., 2023), thereby reducing their effectiveness.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > Paraphrasing (P).
What the sources say
- Partly supportedcites Jain et al., 2023 sourceExperimental — accuracy not yet measured
The Results section reports that paraphrasing prompts significantly reduced attack success and that ChatGPT ignored the adversarial suffix in most cases. The paper tests paraphrasing adversarial prompts, not specifically external content E Tu, so it does not establish that the same defense works in that setting. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, after applying adaptive attacks, the detection rates drop to nearly zero in most cases, demonstrating the effectiveness of adversarial strings in bypassing the detectors.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.1 Detailed Results and Analysis
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show that adaptive attacks consistently achieve success rates above 50% across the targeted defenses and LLM agents—exceeding the ASR before deploying any defense and significantly outperforming non-adaptive attacks.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > Model -level Adversarial finetuning
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the top three strongest defenses shown in Figure 2 for the Vicuna-7B based agent—the fine-tuned detector, sandwich prevention, and adversarial finetuning—as well as defenses specifically targeting adversarial strings, i.e., paraphrasing and perplexity filtering, we observe that the adaptive attack for each defense significantly outperforms other attacks.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.4 Cross Evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Upon closer examination, we found that this is primarily due to many unsuccessful cases in the second step of the data stealing attack.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.1 Detailed Results and Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From Figure 2, we observe that most prompt modification defenses offer weak protection against the attacks for the Vicuna-7B based agent, except for sandwich prevention.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.1 Detailed Results and Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the Vicuna-7B based agent, most defenses reduce the original ASR from 56% to lower values, such as 12% with adversarial finetuning.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 5.2 Experiment Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show that adaptive attacks consistently achieve success rates above 50% across the targeted defenses and LLM agents—exceeding the ASR before deploying any defense and significantly outperforming non-adaptive attacks.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > Model -level Adversarial finetuning
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, a high target rate does not always correspond to a high ASR, as seen in GCG over adversarial finetuning.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 5.2 Experiment Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We present the results in Table 3, showing that our adaptive adversarial strings are more effective than the non-adaptive strings.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.1 Detailed Results and Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 18
- It may state two separate claims as one. 11
Every source the paper cites for this claim was checked and supports it.
In the paper
“AgentDojo (Debenedetti et al., 2024) is a benchmark designed to evaluate IPI attacks on LLM agents within a dynamic simulated environment.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > B Experiments on the AgentDojo Benchmark
What the sources say
- Supportedcites Debenedetti et al., 2024 source
The Introduction describes AgentDojo as a dynamic benchmarking framework for evaluating AI agents when prompt injections are present. It also says agents call tools in a stateful, adversarial environment, supporting the claim's description of a dynamic simulated environment. (LLM, conf=0.94)
plainn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The study of adaptive attacks has also expanded into LLMs, particularly in the context of jailbreak (Jain et al., 2023;Andriushchenko et al., 2024).”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 7.3 Adaptive Attacks
What the sources say
- Supportedcites Jain et al., 2023 source
The Introduction says the paper studies adversarial attacks on LLMs, and Section 4 uses a jailbreaking attack as its testbed. The Abstract and Section 5.3 explicitly discuss adaptive attacks against LLM defenses, supporting the claim that this line of study extends to LLM jailbreaks. (LLM, conf=0.91)
- Supportedcites Andriushchenko et al., 2024 source
The Introduction explicitly says the paper presents model-specific adaptive attacks for evaluating LLM robustness and discusses designing jailbreak attacks. Its “Adaptive attack” section says the authors customize those attacks for each target LLM. This supports the cited point that adaptive-attack research extends to LLM jailbreaking. (LLM, conf=0.94)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As mentioned earlier, the GCG attack over the adversarial training defense achieves a high target rate but a relatively low ASR for the Vicuna-7B based agent.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.1 Detailed Results and Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Using a prefix adversarial string instead of a suffix yields better performance for Llama3-8B.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > A Implementation Details
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The attacker has white-box access to the agent model, defense models, and detailed prompts.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, adaptively trained adversarial strings still bypass these defenses, achieving high ASRs for both agents.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 5.2 Experiment Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, AutoDAN achieves the lowest target rate among the methods but still attains a high ASR.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 5.2 Experiment Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most GCG models achieve high target rates, reflecting their training effectiveness.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 5.2 Experiment Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Although adaptive attacks still yield substantial improvements in ASR, they are not as effective as on the InjecAgent benchmark-where all adaptive attack ASRs exceed 50%.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > B Experiments on the AgentDojo Benchmark
plaindirectn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The attacker has white-box access to the agent model, defense models, and detailed prompts.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this paper, we evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%.”
- Also stated in: § Model -level Adversarial finetuning · boosted
“Our results show that adaptive attacks consistently achieve success rates above 50% across the targeted defenses and LLM agents—exceeding the ASR before deploying any defense and significantly outperforming non-adaptive attacks.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Model -level Adversarial finetuning · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our results indicate that adversarial strings for the Llama3-8B based agent are more challenging to optimize than those for the Vicuna-7B based agent.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > A Implementation Details
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The attacker has white-box access to the agent model, defense models, and detailed prompts.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Llama3-based agent shows greater resilience to IPI attacks, with an original ASR of 9% and even lower ASRs under defenses.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 5.2 Experiment Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe that the fine-tuned detector achieves the highest detection rate on the original IPI attacks for the Vicuna-7B based agent and LLM-based detetor for the Llama3-8B based agent.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.1 Detailed Results and Analysis
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our research is not an exhaustive exploration of all potential defenses that could be adapted for IPI attacks.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 10 Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These longer contexts, which often include multiple rounds of agent-tool interaction, pose greater challenges for optimizing the adversarial string.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > B Experiments on the AgentDojo Benchmark
plaindirectn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: The attacker has white-box access to the agent model, defense models, and detailed prompts.
In the paper
“This aligns with the conclusion in InjecAgent (Zhan et al., 2024) that fine-tuned agents are more resilient to IPI attacks.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 5.2 Experiment Results
What the sources say
- Not checked yetcites Zhan et al., 2024 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This paper presents the first study of defenses and adaptive attacks against LLM agents in the context of IPI attacks.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 7.1 LLM Agent Safety
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“To automate and strengthen these attacks, researchers have developed methods like GCG (Zou et al., 2023) and AutoDAN attacks (Liu et al., 2024a;Zhu et al., 2023), which automatically find prompts to jailbreak LLMs.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 7.2 LLM Safety and Adversarial Attacks
What the sources say
- Not checked yetcites Zou et al., 2023 source
- Not matchedcites Liu et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Zhu et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, attack success can be directly measured by the execution of the attacker tool, in contrast to the uncertain evaluations based on keyword mapping and LLM judgments in previous work.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 7.1 LLM Agent Safety
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our adaptive attacks focus solely on the agent's first-step action.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 10 Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This occurs because all adversarial strings are trained to let the model generate the target output, which only has direct influence for the first step, limiting their impact on the later steps.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.3 Breakdown of ASRs
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This highlights the necessity of conducting adaptive attacks to evaluate the robustness of these defenses.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.4 Cross Evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Originally introduced for jailbreak attacks, GCG aims to train adversarial strings to let the LLM generate affirmative prefixes that induce malicious content following the malicious instruction.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 4 Adaptive Attack Techniques
What the sources say
- Not checked yetcites Zou et al., 2023 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is due to that we use the same model as the agent backbone model in LLM-based detector, and Llama3-8B is strong and better safety aligned, showing better ability in detecting IPI attacks.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.1 Detailed Results and Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We do not account for the combination of defenses.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 10 Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, adaptive attacks in this setting involve crafting adversarial examples in the external content to manipulate the LLM agent.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > Model -level Adversarial finetuning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Among the eight defenses, perplexity filtering and paraphrasing are specifically designed to address adversarial strings.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 6.1 Detailed Results and Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We hypothesize that this discrepancy arises primarily from the significantly longer contexts in AgentDojo, with an average token length of 3,823 and a maximum of 6,152, compared to InjecAgent's average token length of 1,033 and maximum of 1,711.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > B Experiments on the AgentDojo Benchmark
hedgeddirectn = 16- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true.
- The authors state this as an assumption: The attacker has white-box access to the agent model, defense models, and detailed prompts.
In the paper
“Unlike prompt injection attacks in LLMs (Liu et al., 2024b), targeting agents with tool usage poses extra challenges: (1) the attack must compel harmful actions rather than just generating a target output, and (2) the greater complexity of inputs and outputs complicates optimization.”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 7.1 LLM Agent Safety
What the sources say
- Not matchedcites Liu et al., 2024b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“New attacks that bypass existing defenses frequently arise, a phenomenon well-documented in computer vision (Athalye et al., 2018;Tramèr et al., 2020;Yu et al., 2021;Carlini and Wagner, 2017).”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 7.3 Adaptive Attacks
What the sources say
- Not matchedcites Athalye et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Tramèr et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites Carlini and Wagner, 2017 source
Checked before — its earlier verdict is held back until a new check runs.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper relies on this without support.
No studies have yet explored the effectiveness of IPI defenses against adaptive attacks.
“While defenses against IPI attacks have been proposed (Yi et al., 2023), no studies have yet explored their effectiveness against adaptive attacks, leaving their robustness in question.”
The authors state this as an assumption.
The attacker has knowledge of and white-box access to the agent and defenses.
“In the adaptive attack setting, we assume the attacker has knowledge of and white-box access to the agent and defenses (Athalye et al., 2018).”
The authors state this as an assumption.
The attacker has white-box access to the agent model, defense models, and detailed prompts.
“Our attacks assume the attacker has white-box access to the agent model, defense models, and detailed prompts.”
The paper relies on this without support.
This paper is the first to explore adaptive attacks specifically targeting LLM agent safety.
“To the best of our knowledge, this paper is the first to explore adaptive attacks specifically targeting LLM agent safety.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 10 Limitations
“We train all adversarial strings to produce the output: "Thought: I will use the {Attacker tool name} tool to,"”
§ Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents > 5.1 Experiment Settings
“For Llama3-8B based agent, we let target = "{"name": "<T a >"".”
Citations
Cites
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
Carlini, N., & Wagner, D. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140444 · Cited by 966
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024) · arXiv · doi:10.48550/arxiv.2406.13352 · Cited by 616
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
Andriushchenko, M., Croce, F., & Flammarion, N. (2024) · arXiv · doi:10.48550/arxiv.2404.02151 · Cited by 396
Jailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3508 · Cited by 280
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Chen, Z., Xiang, Z., Xiao, C., Song, D., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.12784 · Cited by 246
Research works printed but not linked: 13
- Anish Athalye, Nicholas Carlini, and David A. Wag- ner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Re- search, pages 274-283.
- PMLR.
- Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. 2024. Agent smith: A single image can jailbreak one million multimodal LLM agents exponentially fast. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
- Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
- Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024a. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized eval- uation framework for automated red teaming and ro- bust refusal. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
- Mansi Phute, Alec Helbling, Matthew Hull, Shengyun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2024. LLM self defense: By self examination, llms know they are being tricked. In The Second Tiny Papers Track at ICLR 2024, Tiny Papers @ ICLR 2024, Vienna, Austria, May 11, 2024.
- Yangjun Ruan, Honghua Dong, Andrew Wang, Sil- viu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the risks of LM agents with an lm- emulated sandbox. In The Twelfth International Con- ference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. 2020. On adaptive attacks to ad- versarial example defenses. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024b. GPT-4 is too smart to be safe: Stealthy chat with llms via cipher. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Bar- row, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: Interpretable gradient- based adversarial attacks on large language models. In First Conference on Language Modeling.
Cited by
Sorted by citations
Nasr, M., Carlini, N., Sitawarin, C., Schulhoff, S., Hayes, J., Ilie, M., Pluto, J., Song, S., Chaudhari, H., Shumailov, I., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.09023 · Cited by 151
Prompt Flow Integrity to Prevent Privilege Escalation in LLM AgentsPreprint
Kim, J., Choi, W., & Lee, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.15547 · Cited by 99
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74
From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
Ferrag, M. A., Tihanyi, N., Hamouda, D., Maglaras, L., Lakas, A., & Debbah, M. (2026) · ICT Express · doi:10.1016/j.icte.2025.12.001 · Cited by 46
Agent Skills Enable a New Class of Realistic and Trivially Simple Prompt InjectionsPreprint
Schmotz, D., Abdelnabi, S., & Andriushchenko, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.26328 · Cited by 41
The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to AnalysisPreprint
Wang, P., Li, X., Xiang, C., Zhang, J., Li, Y., Zhang, L., Wang, X., & Tian, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.10453 · Cited by 40
Arunkumar, V., R, G. G., & Buyya, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.12560 · Cited by 37
Don't Let the Claw Grip Your Hand: A Security Analysis and Defense Framework for OpenClawPreprint
Shan, Z., Xin, J., Zhang, Y., & Xu, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.10387 · Cited by 36
STAC: When Innocent Tools Form Dangerous Chains for LLM Agents
Li, J.-J., He, J., Shang, C., Kulshreshtha, D., Xian, X., Zhang, Y., Su, H., Swamy, S., & Qi, Y. (2026) · Workshop for Research on Agent Language Models (REALM) · doi:10.18653/2026.realm-1.17 · Cited by 33
ChatInject: Abusing Chat Templates for Prompt Injection in LLM AgentsPreprint
Chang, H., Jun, Y., & Lee, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.22830 · Cited by 31
Free-MAD: Consensus-Free Multi-Agent Debate
Cui, Y., Fu, H., Zhang, H., Wang, L., & Zuo, C. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1600 · Cited by 28
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28
MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent SkillsPreprint
Guo, W., Zeng, W., Liu, C., Jia, X., Xu, Y., Tang, L., Fang, Y., & Liu, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.07131 · Cited by 28
Mao, Y., Cui, T., Liu, P., You, D., & Zhu, H. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3856301 · Cited by 25
What Makes a Good LLM Agent for Real-world Penetration Testing?Preprint
Deng, G., Liu, Y., Li, Y., Yang, R., Xie, X., Zhang, J. M., Qiu, H., & Zhang, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.17622 · Cited by 24
System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User InjectionPreprint
Li, Z., Guo, J., & Cai, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.06493 · Cited by 21
The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI SocietiesPreprint
Wang, C., Li, C., Liu, S., Chen, Z., Hou, J., Qi, J., Li, R., Zhang, L., Ye, Q., Liu, Z., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.09877 · Cited by 21
Qi, J., Li, M., Liu, J., Shu, Y., Yu, D. S. F., Ma, S., Cui, W., Zhao, Y., Chen, Y., Jiang, R., et al. (2026) · Academia AI and Applications · doi:10.20935/acadai8260 · Cited by 20
Lu, Y., Zhao, M., Ju, T., Yan, Z., Ma, X., Guo, Y., Ding, D., Liu, G., & Zhang, Z. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1230 · Cited by 19
Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility TreePreprint
Johnson, S., Pham, V., & Le, T. (2025) · arXiv (Cornell University) · Cited by 19
Alamsabi, M., Tchuindjang, M., & Brohi, S. N. (2026) · Algorithms · doi:10.3390/a19010092 · Cited by 16
ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time CorrectionPreprint
Wang, C., Zhang, F., Zhang, J., Zhang, Z., Wang, Y., Huang, L., Gao, J., Chen, Z., & Lim, W. Y. B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.20708 · Cited by 16
Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment
Xie, Y., Luo, M., Liu, Z., Zhang, Z., Zhang, K. F., Liu, Y., Tao, C., Wang, C., Li, Z., Chen, P., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832267 · Cited by 15
Hidden-in-Plain-Text: A Benchmark for Social-Web Indirect Prompt Injection in RAGPreprint
Guo, H., & Wei, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.10923 · Cited by 14
FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web AgentsPreprint
Kerboua, I. L., Shayegan, S. O., Thakkar, M., Lù, X. H., Boisvert, L., Caccia, M., Espinas, J., Aussem, A., Églin, V., & Lacoste, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.03204 · Cited by 13
Assessing Automated Prompt Injection Attacks in Agentic EnvironmentsPreprint
Hofer, D., Debenedetti, E., & Tramèr, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.10525 · Cited by 13
Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense FrameworksPreprint
Ji, Z., Wang, X., Li, Z., Ma, P., Gao, Y., Wu, D., Yan, X., Tian, T., & Wang, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.15203 · Cited by 12
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11
From Texts to Shields: Convergence of Large Language Models and CybersecurityPreprint
Li, T., Yang, Y.-T., Pan, Y., & Zhu, Q. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.00841 · Cited by 11
ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplification
Lin, X., Li, P., Zeng, Z., Li, T., Wei, T., Ning, X., Li, G., Chen, Y., & Tong, H. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.799 · Cited by 7
Hatami, M., Pham, V. T., Lakadawala, H., & Chen, Y. Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.20184 · Cited by 7
Securing educational LLMs: A generalised taxonomy of attacks on LLMs and DREAD risk assessment
Zahid, F., Sewwandi, A., Brandon, L., Kumar, V., & Sinha, R. (2025) · High-Confidence Computing · doi:10.1016/j.hcc.2025.100371 · Cited by 6
Johnson, S., Pham, V., & Le, T. (2025) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2025.emnlp-demos.55 · Cited by 6
Hybrid Real-time Framework for Detecting Adaptive Prompt Injection Attacks in Large Language Models
Prakash, C., Lind, M. L., & Cruz, E. D. L. (2026) · Journal of Computing Theories and Applications · doi:10.62411/jcta.15254 · Cited by 6
AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal AgentsPreprint
Li, Y., Cao, Y., Wang, D., & Xiao, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.04257 · Cited by 6
Prismata: Confining Cross-Site Prompt Injection in Web AgentsPreprint
Villa, C., Ozdarendeli, A. E., Tan, S., & Popa, R. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.08147 · Cited by 6
Khan, A. R., AlKhanbashi, K., & Monem, A. A. (2026) · Computers · doi:10.3390/computers15090570 · Cited by 5
Enhancing security in LLM applications: a performance evaluation of early detection systems
Gakh, V., & Bahşi, H. (2026) · International Journal of Information Security · doi:10.1007/s10207-026-01338-7 · Cited by 4
Soft Instruction De-escalation DefensePreprint
Walter, N. P., Sitawarin, C., Hayes, J., Stutz, D., & Shumailov, I. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.21057 · Cited by 4
QueryIPI: Query-agnostic Indirect Prompt Injection on Coding AgentsPreprint
Xie, Y., Liu, Z., Luo, M., Zhang, Z., Zhang, K., Yuan, Y., Li, Z., Chen, P., Wang, S., & She, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.23675 · Cited by 4
ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code CommentsPreprint
Wu, Z., & Nita-Rotaru, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.24964 · Cited by 4
EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing AgentsPreprint
Li, N., Ma, Y., Cao, Y., Xiang, C., Suh, G. E., Li, B., Choi, Y., Song, D., & Xiao, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.05903 · Cited by 4
VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language ModelsPreprint
Sadanandan, B., & Behzadan, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.00052 · Cited by 3
DRAFT: Task Decoupled Latent Reasoning for Agent SafetyPreprint
Wang, L., Fang, J., Zhang, D., Shen, F., Wang, X. G., & Chua, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.03242 · Cited by 3
Agent Hacks Agents: Autoresearch Discovers Vulnerabilities in Production AgentsPreprint
Mao, X., Qian, R., Zheng, X., & Wang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.11698 · Cited by 3
Safety in Self-Evolving Agents: A SurveyPreprint
Chen, J., Feng, Z., Ma, O., Yan, Y., Lin, R., Zhang, H., Du, L., An, H., Yang, Y., Liu, J., et al. (2026) · arXiv (Cornell University) · doi:10.13140/rg.2.2.23319.74403 · Cited by 3
Securing Tool-Using AI Agents Against Injection and Authority Misuse
Kanaker, H., Fakhouri, H. N., Karim, N. A., Abuhamdeh, M., Ismail, N. H. A., & Fakhouri, S. N. (2026) · Computation · doi:10.3390/computation14050098 · Cited by 2
Bypassing AI Control Protocols via Agent-as-a-Proxy AttacksPreprint
Isbarov, J., & Kantarcıoğlu, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.05066 · Cited by 2
From Thinker to Society: Security in Hierarchical Autonomy Evolution of AI AgentsPreprint
Zhang, X., Zhou, L., Xu, X., Wu, J., Du, T., Huang, H., Peng, H., & Liu, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.07496 · Cited by 2
Shen, Y., Pan, X., Hong, G., & Yang, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.24203 · Cited by 2
GIF: Locally Sound Geometric Information Flow Control for LLMsPreprint
Storek, A., Holzer, N., Zhang, Z., & Jana, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.23277 · Cited by 2
Rahman, M. H., & Kim, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.16032 · Cited by 2
Zhang, B., Xiao, Q., Dang, L., & Wu, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.07529 · Cited by 2
Investigating Semantic Drift in GPT-4 Following Prompt Injection Attacks
Potti, A. (2026) · American journal of student research. · doi:10.70251/hyjr2348.41482498 · Cited by 1
Secure Prompt Engineering Patterns for Cloud LLM Agents
Sudarsan, S., Patel, A., Upadhyay, C., Gudur, V., & Matam, P. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395764 · Cited by 1
Safety Evaluation for Large Language Models with Metamorphic Testing: An Empirical Study
Xing, Y., Huang, J., Zhao, H.-J., Zhang, L., & Li, W.-J. (2026) · Journal of Computer Science and Technology · doi:10.1007/s11390-026-5705-z · Cited by 1
TRACER: Trace-Regularized Training Against Indirect Prompt Injection
Mishra, K., Lahlou, S., & Lukas, N. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.2124 · Cited by 1
CourtGuard: A Local, Multiagent Prompt Injection ClassifierPreprint
Wu, I., & Maslowski, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.19844 · Cited by 1
Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM AgentsPreprint
Ruan, J., Li, P., Chen, Y., Li, F., & Feng, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.12977 · Cited by 1
The Claws in Plain Sight: Unauthorized Context Disclosure through LLM Agent Tool CallsPreprint
Dong, B., Guo, Z., Lu, T., & Wang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.20658 · Cited by 1
When Context Gets Root: Privilege Escalation in LLM HarnessesPreprint
He, X., Chen, Y., Qian, Y., Wei, H., Chen, L., Fu, Z., Wang, L., Wu, H., & Mao, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.27299 · Cited by 1
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework
Ji, Z., Wu, D., Jiang, W., Ma, P., Li, Z., Gao, Y., Wang, S., & Li, Y. (2026) · arXiv · doi:10.48550/arxiv.2601.11893
Rojas-Ordoñez, S., Segura, M., Mendoza, V., Fernández‐Gámiz, U., & Zulueta, E. (2025) · Applied Sciences · doi:10.3390/app152412986
Modern Day LLMs : A GOLDMINE or a LANDMINE ?
S, A. R., Muthudhanush, A. R., & D, S. M. (2025) · International Conference on Intelligent Systems and Pioneering Innovations in Robotics and Electric Mobility (INSPIRE) · doi:10.1109/inspire67328.2025.11300550
ReIn: Conversational Error Recovery with Reasoning InceptionPreprint
Kim, T., Nam, J., Basu, C., Fan, X., Ma, C., Ji, H., Tür, G., & Hakkani-Tur, D. Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.17022
A Comprehensive Assessment Tool for Prompt Injection Attacks in Large Language Models (LLMs)
Dao, A., VanGorder, R., Gebril, M., & Abdelhamid, S. E. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395844
Jailbreaking Large Language Models: Techniques, Trends, Defenses, and Open Challenges
Vinay, V. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395786
DHR-LLM: A Dynamic Heterogeneous Redundancy Framework for Resilient Large Language Model Inference
Chen, P., Yu, Z., & Zhang, S. (2025) · International Conference on Cyber Resilience and Endogenous Safety & Security (CRESS) · doi:10.1109/cress68073.2025.11452519
Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection DetectionPreprint
Munirathinam, T. (2026) · arXiv (Cornell University) · doi:10.5281/zenodo.19644135
EvoShield: Selective Test-Time Adaptation for Prompt Injection Detection via Active LLM Querying
郑赞红, Liang, J., Hu, M., Pei, Y., Guobao, X., & Zhenlu, W. (2026) · Mathematics · doi:10.3390/math14101719
RIPE-II: Retrieval In-Place Poisoning Evaluation with Indirect Injections
Malladi, G., & Thamilarasu, G. (2026) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn69566.2026.00078
PROMPT INJECTION RESISTANCE IN TOOL-USING LLM APPLICATIONS: ARCHITECTURAL PATTERNS AND FAILURE MODES
Andrey, Z. (n.d.) · CyberLeninK (CyberLeninka)
Veritas: Dynamic Evaluation Framework for Rag Systems' Defense Against Prompt Injection Attacks
John, A., & Thamilarasu, G. (2026) · International Conference on Smart Applications, Communications and Networking (SmartNets) · doi:10.1109/smartnets69662.2026.11604866
Self-State Attacks on Self-Hosted AI Agents: How Far Can OS Defenses Go?Preprint
Chen, Y., Denis, N., Pietro, R. D., & Schmidhuber, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.17986
PlanFlip: Attacking Multi-Agent LLM Systems via Planning-Phase Prompt InjectionPreprint
Wang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.16199
GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models
Zeng, L., Ye, Z., Xie, M., Zhang, H., Wang, X., Li, Y., & Li, Z. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835361
Safety Claims Under Pressure: A Methodological Review of LLM and MLLM Jailbreaks
Ma, W., Ren, W., & Zhang, Y. (2026) · International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA) · doi:10.1109/caibda70336.2026.11621495
Gao, W. (2026) · Stout in Computer Science and Technology Studies · doi:10.61424/hewtat90
Harmful Content Is Not Enough: Continuation Framing Moderates In-Context Emergent MisalignmentPreprint
Liu, P., Wang, X., Cui, Z., Liang, D., & Ye, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.08212
SoK: Security Vulnerabilities in Agentic AI Systems
Guan, S., Kovalev, A., & Xie, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35579-9_6
Agentic AI Security in IoMT: A PRISMA-Guided Scoping Review and Layered Threat–Defense Taxonomy
Abylkan, B., Serik, D., Mussa, A., Sapash, S., Kurakbayev, A., & Toksanovna, A. S. (2026) · International Conference on Cybersecurity, Digital Forensics, and AI Applications (ICCSDFAI) · doi:10.1109/iccsdfai70505.2026.11647867
A Formal Framework of Architectural Intent Collapse for Tool-Level Attacks on LLM Agents
Feng, Z., Liu, Z., Ma, M., Zhuang, D., & Gao, J. (2026) · Electronics · doi:10.3390/electronics15163739
Chen, J., & Lu, R.-S. (2026) · IEEE Access · doi:10.1109/access.2026.3726909
Garugu, J. S. S. (2026) · International Journal of Versatile Research and Analysis · doi:10.56975/ijvra.v4i8.710307
Rahman, M. H., & Kim, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.27092
ROUSEK, A. (2026) · Challenges to national defence in a contemporary geopolitical situation · doi:10.47459/cndcgs.2026.16
Indirect Prompt Injection in Municipal Document-Processing Copilots: Attacks, Defences and Harm
Cisneros-González, J., Ondiviela, J. A., & Sánchez-Soriano, J. (2026) · Street Art & Urban Creativity · doi:10.62161/sauc.v12.6399
IBBench-Light: A Paired Evaluation of Task-Conditioned Responses to External DirectivesPreprint
Zhou, K., Li, Z., Janet, S., Qian, G., & Xiao, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.13725
Cao, X., & Bao, Y. (2026) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia71496.2026.11684636
Optimized Multi-Agent Defense Pipeline Against Advanced Prompt Injection Attacks
Pearson, B. (2026) · ResearchWorks at the University of Washington (University of Washington)
Szczepaniak, J., Feldman, E., Viner, N., & Nassi, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.22510
Chen, G., Xia, Y., Ma, X., & Li, Z. M. (2026) · Symmetry · doi:10.3390/sym18101644
Self-Evolving Defense: Continual Security Policy Learning for LLM AgentsPreprint
Le, M. N., Gondi, N., Peng, Y., Ni, R., Jia, L., Chen, B., & Zheng, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.36603
Can Agents Trust Their Skills? Uncovering Unsafe Chains of Trust in Skill-Based LLM AgentsPreprint
Wang, Y., Zhang, Z., Chen, K., Chen, K., Zhang, Y., Ma, D., Dai, J., & Sun, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.39065
AgentBreaker: Evaluating Context-Aware Indirect Prompt Injection Risks in Modern Web Agents
Son, Y., Lee, C., Shin, D., Lee, B., Hong, S., & Son, S. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832165
ActionGuard: Tool Call Authorization under Poisoned SkillsPreprint
Han, J., Jang, Y., Kwak, B. I., & Han, M. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.39450
Qayyum, I., Asghar, B., Ahmed, T., Zainab, M., & Ullah, R. S. (2026) · Annual Methodological Archive Research Review
Metadata
- Authors
- Zhan, Q., Fang, R., Panchal, H. S., & Kang, D.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- ORCID
- Kang, Daniel
- Venue
- Findings of the Association for Computational Linguistics: NAACL 2025
- Published
- 2025
- Pages
- 7101-7117
- OpenAlex
- W4411119880
- Cited by
- 124
- References
- 4
- Citations by year
- 2026: 103 · 2025: 21
- FWCI
- 44.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Keywords
- LLM agents, adaptive attacks, defense robustness, external tool integration, security vulnerabilities, attack evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
