AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases

The optimized trigger is resilient to diverse augmentations or perturbations.

“Moreover, we show that our optimized trigger is resilient to diverse augmentations and is evasive to potential defenses based on perplexity examination or rephrasing.”

• Also stated in: § How does each individual loss contributes to AGENTPOISON? · plain

  “AGENTPOISON is resilient to perturbations in the trigger sequence.”

The optimized trigger is evasive to perplexity-examination defenses.

“Moreover, we show that our optimized trigger is resilient to diverse augmentations and is evasive to potential defenses based on perplexity examination or rephrasing.”

• Also stated in: § How does AGENTPOISON perform under potential defense? · plain

  “Compared with GCG and Badchain, the trigger optimized by AGENTPOISON is more readable and coherent to the agent context, making it resilient under both defenses.”

• cites [2]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

AGENTPOISON achieves 79.0% ASR-r when the trigger contains only one token.

“Meanwhile, it also achieves 79.0%ASR-r when the trigger only contains one token.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

AGENTPOISON achieves an average attack success rate of at least 80% on each of the three tested agents.

“On each agent, AGENTPOISON achieves an average attack success rate of ≥ 80% with minimal impact on benign performance (≤ 1%) with a poison rate < 0.1%.”

• Also stated in: § Query encoder · plain

  “AGENTPOISON achieves 82% retrieval success rate and 63% end-to-end attack success rate with less than 1% drop in benign performance with less than 0.1% poisoning ratio.”

• Also stated in: § 4.2 Result · plain

  “AGENTPOISON has minimal impact on benign performance of average 0.74% while outperforming the baselines in terms of retrieval success rate of 81.2% in average, while an average 59.4% generates target actions where 62.6% result in actual target impact to the environment.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

A trigger optimized for one RAG embedder can be transferred to attack other RAG embedders.

“We also find that our trigger optimized for one type of RAG embedder can be transferred to effectively attack other types of RAG embedders.”

• Also stated in: § 4.2 Result · boosted

  “AGENTPOISON has high transferability across embedders.”

• cites [14]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The attacker has white-box access to the victim agent's RAG embedder for trigger optimization.

The optimized trigger transfers well across different RAG embedders.

“the optimized backdoor trigger exhibits superior transferability, in-context coherence, and stealthiness.”

• Also stated in: § 4.2 Result · boosted

  “AGENTPOISON has high transferability across embedders.”

• Also stated in: § 4.2 Result · plain

  “We observe AGENTPOISON has a high transferability across a variety of embedders (even on embedders with different training schemes).”

• cites [14]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The attacker has white-box access to the victim agent's RAG embedder for trigger optimization.

AGENTPOISON achieves an 82% retrieval success rate.

“AGENTPOISON outperforms baseline attacks by achieving 82% retrieval success rate and 63% end-to-end attack success rate with less than 1% drop in the benign performance and with poisoning ratio less than 0.1%.”

• Also stated in: § 4.2 Result · plain

  “AGENTPOISON has minimal impact on benign performance of average 0.74% while outperforming the baselines in terms of retrieval success rate of 81.2% in average, while an average 59.4% generates target actions where 62.6% result in actual target impact to the environment.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

AGENTPOISON maintains high benign utility, ACC at least 90%, regardless of the number of poisoned instances or trigger tokens.

“Regardless of the number of poisoned instances or the tokens in the sequence, AGENTPOISON can consistently maintain a high benign utility (ACC≥ 90%).”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

AGENTPOISON is slightly compromised by letter injection perturbations.

“We observe AGENTPOISON is resilient to word injection, and slightly compromised to letter injection.”

The optimized trigger has high in-context coherence.

“the optimized backdoor trigger exhibits superior transferability, in-context coherence, and stealthiness.”

• Also stated in: § How does each individual loss contributes to AGENTPOISON? · plain

  “Besides, while adding L coh slightly degrades the performance, it leads to better in-context coherence, which can effectively bypass some perplexity-based countermeasures.”

AGENTPOISON achieves high ASR-r, 62.0% on average, when only one instance is poisoned.

“We observe that after optimization, AGENTPOISON has high ASR-r (62.0% in average) when we only poison one instance in the database.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

The optimized trigger is evasive to rephrasing defenses.

“Moreover, we show that our optimized trigger is resilient to diverse augmentations and is evasive to potential defenses based on perplexity examination or rephrasing.”

• Also stated in: § How does AGENTPOISON perform under potential defense? · plain

  “Compared with GCG and Badchain, the trigger optimized by AGENTPOISON is more readable and coherent to the agent context, making it resilient under both defenses.”

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The uniqueness loss L_uni significantly contributes to high ASR-r in AGENTPOISON.

“We observe L uni significantly contributes to the high ASR-r in AGENTPOISON while ACC is more sensitive to L cpt where more concentrated qt generally lead to better ACC.”

AGENTPOISON outperforms four baseline attacks on three types of LLM agents.

“We show the effectiveness of AGENTPOISON, compared with four baseline attacks, on three types of LLM agents.”

• Also stated in: § 4.2 Result · boosted

  “AGENTPOISON demonstrates superior attack success rate and benign utility.”

• Also stated in: § 5 Conclusion · boosted

  “Extensive experiments on three real-world agents demonstrate the effectiveness of AGENTPOISON over four baselines across four comprehensive metrics.”

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [39]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

AGENTPOISON achieves a 63% end-to-end attack success rate.

“AGENTPOISON outperforms baseline attacks by achieving 82% retrieval success rate and 63% end-to-end attack success rate with less than 1% drop in the benign performance and with poisoning ratio less than 0.1%.”

• Also stated in: § 4.2 Result · plain

  “AGENTPOISON has minimal impact on benign performance of average 0.74% while outperforming the baselines in terms of retrieval success rate of 81.2% in average, while an average 59.4% generates target actions where 62.6% result in actual target impact to the environment.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

This unique-region design gives AGENTPOISON high ASR even when only one instance is injected with a single-token trigger.

“Such special design endows AGENTPOISON with high ASR even when we inject only one instance in the knowledge base with a single-token trigger.”

• Also stated in: § ASR-r ASR-a ASR-t ACC ASR-r ASR-a ASR-t ACC ASR-r ASR-a ASR-t ACC · plain

  “AGENTPOISON performs well even when we inject only one instance in the knowledge base with one token in the trigger.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

AGENTPOISON uses a poison rate below 0.1% of the original database instances.

“On each agent, AGENTPOISON achieves an average attack success rate of ≥ 80% with minimal impact on benign performance (≤ 1%) with a poison rate < 0.1%.”

• Also stated in: § Query encoder · plain

  “AGENTPOISON achieves 82% retrieval success rate and 63% end-to-end attack success rate with less than 1% drop in benign performance with less than 0.1% poisoning ratio.”

• Also stated in: § A.1.1 Hyperparameters · plain

  “We inject fewer than 0.1% instances w.r.t. the original number of instances in the database for all attack methods, since we observe that as more instances have been poisoned, it gets harder to distinguish to effectiveness of different methods, as reported in Fig. 4.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

AGENTPOISON is resilient to word injection perturbations.

“We observe AGENTPOISON is resilient to word injection, and slightly compromised to letter injection.”

Spurious correlation poisoning is more stealthy than adversarial backdoor because it leaves the original action plan unchanged.

“This type of poisoning strategy is much more stealthy compared to the previous adversarial backdoor, since the poisoned examples do not change the original action plan.”

Almost all baseline attacks achieve high attack success rate on EHRAgent's original small memory database.

“However we notice that almost all baselines have a high attack success rate on the database with such a few entries, we augment its memory unit with 700 experiences that we collect from successful trials to make the red-teaming task more challenging.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

AGENTPOISON is effective in optimizing triggers to achieve high retrieval accuracy and attack success rate.

“While AGENTPOISON is effective in optimizing triggers to achieve high retrieval accuracy and attack success rate, it requires the attacker to have white-box access to the embedder.”

AGENTPOISON queries are highly evasive because their perplexity distribution is inseparable from benign queries.

“Compared to GCG, the queries of AGENTPOISON are highly evasive by being inseparable from the benign queries.”

Triggers from AgentPoison usually transfer effectively between dense retrievers.

“trigger optimized with AGENTPOISON generally transfer well across dense retrievers”

Rephrasing the trigger while preserving its semantics maintains high attack performance.

“Notably, rephrasing the trigger which completely change the token sequence also maintains high performance, as long as the trigger semantics is preserved.”

Transfer is stronger between embedders that share a training strategy, whether end-to-end (REALM, ORQA) or contrastive (DPR, ANCE, BGE).

“triggers transfer better among embedders with similar training strategy (i.e.end-to-end (REALM, ORQA); contrastive (DPR, ANCE, BGE)).”

Current LLM attacks such as jailbreaking and in-context learning backdooring cannot effectively attack RAG-based LLM agents.

“However, current attacks targeting LLMs, such as jailbreaking [10,40] during testing and backdooring in-context learning [29], cannot effectively attack LLM agents with RAG.”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

State-of-the-art LLMs generate undesired adversarial responses when given malicious demonstrations during knowledge-enabled reasoning.

“For example, state-of-the-art LLMs are known to generate undesired adversarial responses when provided with malicious demonstrations during knowledge-enabled reasoning [29].”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

AGENTPOISON is the first backdoor attack aimed at generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base.

“we propose a novel red teaming approach AGENTPOISON, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base.”

• Also stated in: § Query encoder · plain

  “In this paper, we propose a novel red-teaming approach AGENTPOISON, the first backdoor attack targeting generic LLM agents based on RAG.”

• Also stated in: § Red-teaming LLM Agents · hedged

  “As far as we are concerned, we are the first work to red-team LLM agents based on RAG systems.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

Backdoor attacks such as BadChain use suboptimal triggers that fail to guarantee retrieval of malicious demonstrations in LLM agents, leading to unsatisfactory attack success rates.

“Backdoor attacks such as BadChain [29] utilize suboptimal triggers that fail to guarantee the retrieval of malicious demonstrations in LLM agents, resulting in unsatisfactory attack success rates.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Jailbreaking attacks like GCG face challenges against RAG agents because retrieval diversity can mitigate injected adversarial suffixes.

“Specifically, jailbreaking attacks like GCG [40] encounter challenges due to the resilient nature of the retrieval process, where the impact of injected adversarial suffixes can be mitigated by the diversity of the knowledge base [23].”

• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

AGENTPOISON does not need extra model training or fine-tuning.

“Unlike conventional backdoor attacks, AGENTPOISON requires no additional model training or fine-tuning, and the optimized backdoor trigger exhibits superior transferability, in-context coherence, and stealthiness.”

• Also stated in: § 3.3.1 Overview · plain

  “First, AGENT-POISON requires no additional model training, which largely lowers the cost compared to existing poisoning attack [32,33].”

• Also stated in: § 5 Conclusion · plain

  “Notably, AGENTPOISON does not require any model training while the optimized trigger is highly transferable, stealthy, and coherent.”

Letter injection can change over three tokens in the sequence, which can completely flip the semantic distribution of the trigger.

“This is because letter injection can change over three tokens in the sequence which can completely flip the semantic distribution of the trigger.”

AGENTPOISON's objective function maps triggered instances into a unique region in the RAG embedding space, separating them from benign instances.

“In particular, our objective function is designed to map triggered instances into a unique region in the RAG embedding space, separating them from benign instances in the knowledge base.”

• Also stated in: § 3.3.1 Overview · plain

  “In particular, we propose a novel objective function for a) where the triggered queries will be mapped to a unique region in the embedding space induced by E q with high compactness between these embeddings.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

An adversary can induce an LLM agent to produce malicious outputs or actions by compromising its memory and RAG so that malicious demonstrations are more easily retrieved.

“Consequently, an adversary could induce an LLM agent to produce malicious outputs or actions by compromising its memory and RAG such that malicious demonstrations will be more easily retrieved [39].”

• cites [39]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

AGENTPOISON requires the attacker to have white-box access to the embedder.

“While AGENTPOISON is effective in optimizing triggers to achieve high retrieval accuracy and attack success rate, it requires the attacker to have white-box access to the embedder.”

• AssumptionThe authors state this as an assumption: The attacker has white-box access to the victim agent's RAG embedder for trigger optimization.

By making all retrieved demonstrations poisoned, AgentPoison can bypass the state-of-the-art defense.

“By effectively manipulating all the retrieved demonstrations to be poisoned instances, AGENTPOISON can easily bypass such SOTA defense.”

The defense method in [28] can hardly defend against AGENTPOISON because AGENTPOISON can ensure all retrieved instances are poisoned.

“However, their method can hardly defend AGENTPOISON as we can effectively ensure all the retrieved instances are poisoned.”

• cites [28]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

The target generation loss can be efficiently approximated using finite samples with polynomial complexity.

“While Eq. ( 9) only works for white-box LLMs, we can efficiently approximate L tar (x t ) using finite samples with polynomial complexity.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

With enough samples, the finite-sample estimate of the target constraint approaches L_tar at a polynomial rate and with high probability.

“Therefore, the finite-sample approximation of the target constraint function converges polynomially (to 1/ϵ) to L tar with high probability as the number of samples increases.”

A test query containing the trigger will fall into the same embedding cluster, causing retrieval of malicious key-value pairs.

“Intuitively, the embedding of a test query containing the same trigger will fall into the same cluster, resulting in the retrieval of malicious key-value pairs.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

The AGENTPOISON trigger can be injected into any position of the query, unlike suffix optimization.

“Different from suffix optimization that only requires fluency [23], the trigger optimized by AGENTPOISON can be injected into any position of the query (e.g. between two sentences).”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

Minimizing the compactness loss further reduces the poisoning ratio.

“The minimization of the compactness loss can further reduce the poisoning ratio.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

AGENTPOISON can ensure all retrieved instances are poisoned.

“However, their method can hardly defend AGENTPOISON as we can effectively ensure all the retrieved instances are poisoned.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

Conclusions from red-teaming LLMs or RAG individually do not transfer well to LLM agents, which are more complex systems.

“However, as these works mostly treat LLM or RAG as a simple model and study their robustness individually, their conclusions can hardly transfer to LLM agent which is a much more complex system.”

LLM agents typically use a single encoder to map both queries and keys into an embedding space.

“Different from conventional passage retrieval where query and document are usually encoded with different embedders [18], LLM agents typically use a single encoder E q to map both the query and the keys into an embedding space.”

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

AGENTPOISON can transfer well among different embedders even with different training schemes because it optimizes for a semantically unique region in the embedding space.

“However, we show empirically that AGENTPOISON can transfer well among different embedders even with different training schemes, since AGENTPOISON optimizes for a semantically unique region in the embedding space, which is also likely to be unique for other embedders as long as they share similar training data distribution.”

Directly maximizing the attacker's two objectives with gradient-based methods is challenging because the trigger affects both retrieval and target action generation in the complex RAG procedure.

“However, directly maximizing Eq. ( 1) and Eq.(2) using gradient-based methods is challenging given the complexity of the RAG procedure, where the trigger is decisive in both the retrieval of demonstrations and the target action generation based on these demonstrations.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

An attacker can red-team a proprietary agent by using a public open-source embedder to optimize a universal trigger.

“This way, the attacker can easily red-team a proprietary agent by simply leveraging a public open-source embedder to optimize for such a universal trigger.”

High ASR-r and ACC of AGENTPOISON can be attributed to its optimization objective.

“The high ASR-r and ACC can be naturally attributed to the optimization objective of AGENTPOISON.”

• AssumptionThe authors state this as an assumption: The attacker has partial access to the victim agent's RAG database and can inject a small number of malicious instances.

Through the spurious correlation backdoor, the agent might learn to link the trigger with the target action.

“where the agent may be misled to associate the target action with the trigger via this backdoor.”

For black-box LLMs, AgentPoison can use a finite-sample indicator to approximate L_tar.

“However, AGENTPOISON can be adapted for black-box LLMs setting by approximating L tar (x t ) via the following finite-sample indicator function.”