Dissecting Adversarial Robustness of Multimodal LM Agents

When the policy model remains uncompromised but the value function is attacked, we observe an ASR of 8% in Figure 6(B), solely caused by the value function.

“In this scenario, we observe an ASR of 8% in Figure 6(B), solely caused by the value function.”

The reflexion agent suffers from a 20% relative increase in ASR compared to the base agent if the evaluator and the policy model are jointly attacked.

“the reflexion agent suffers from a 20% relative increase in ASR compared to the base agent if the evaluator and the policy model are jointly attacked.”

A more aggressive prompt asking the model to immediately output a stop action when it observes inconsistencies or adversarial instructions still fails to improve robustness.

“We then try a more aggressive prompt by asking the model to immediately output a stop action when it observes inconsistencies or adversarial instructions.However, Figure 7 (2nd bar) shows that this still fails to improve robustness.”

If both the value function and the policy model are both attacked, an attacked value function increases the ASR from 31% to 38%.

“Figure 6(D) shows that an attacked value function increases the ASR from 31% to 38%.”

The captioner attack still achieves a 31% ASR.

“Figure 4(B) shows that the captioner attack still achieves a 31% ASR.”

With text access, the prompt injection attack on a GPT-4o-based policy model achieves an ASR of 40%.

“With text access, the prompt injection attack on a GPT-4o-based policy model achieves an ASR of 40% (Figure 4(A)).”

• AssumptionThe authors state this as an assumption: The attacker cannot manipulate the user goal or the agent (e.g., prompts, model parameters) directly.
• AssumptionThe authors state this as an assumption: The attacker can only access a limited part of the environment (the untrusted part), and cannot modify the trusted part.

Optimizing the image at 180px is important for the CLIP attack.

“We find that optimizing the image at 180px is important for the CLIP attack.”

92% of captions successfully incorporate the adversarial text (λ is 0.92 on the edge from captioner to policy model).

“Notably, 92% of captions successfully incorporate the adversarial text (λ is 0.92 on the edge from captioner to policy model).”

The success rate of the CLIP attack does not change with the inference resolution.

“On the other hand, the success rate does not change with the inference resolution”

The paraphrase defense, where the untrusted text input to the LM is paraphrased by GPT-4o, can slightly lower the ASR from 31% to 27.5%.

“We see that the paraphrasing defense can slightly lower the ASR from 31% to 27.5%.”

• cites Jain et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Current state-of-the-art agents, including those using GPT-4o in advanced frameworks such as reflexion and tree search, are highly susceptible even to black-box attacks.

“We find that current state-of-the-art agents -including those using GPT-4o in advanced frameworks such as reflexion and tree search -are highly susceptible even to black-box attacks.”

Attacking the evaluator alone can manipulate the reflexion agent.

“Figure 5( B) shows that attacking the evaluator alone can manipulate the reflexion agent.”

The system prompt defense (Hines et al., 2024), which encourages the backbone LM to prioritize visual inputs and ignore adversarial instructions, fails to improve robustness over the baseline without defense.

“Figure 7 (1st bar) shows that this fails to improve the robustness over the baseline without defense.”

• cites Hines et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Commonly used components in agents, evaluators and value functions, can decrease robustness in the worst-case scenario.

“We demonstrated how commonly used components in agents -evaluators and value functions -can decrease the robustness in the worst-case scenario”

Lower optimization resolution improves the CLIP attack (leads to higher success).

“Lower optimization resolution improves the CLIP attack.”

• Also stated in: § Additional information · plain

  “We see that lower optimization resolution leads to higher success”

When the evaluator is not attacked, it provides a 23% relative reduction in attack success by rejecting adversarial actions and providing reflections.

“when the evaluator is not attacked, it provides a 23% relative reduction in attack success by rejecting adversarial actions and providing reflections.”

We observe a positive correlation between ASR and benign SR across models and settings.

“In general, we observe a positive correlation between ASR and benign SR across models and settings.”

GPT-4o is much more robust than GPT-4V, with ASR 31% vs 67%.

“In §5.1, we have seen that GPT-4o is much more robust than GPT-4V (ASR 31% vs 67%).”

In this scenario, the reflexion agent becomes less robust than the base agent-the ASR increases from 31% of the base agent to 36% of the reflexion agent with an attacked evaluator.

“Interestingly, in this scenario, the reflexion agent becomes less robust than the base agent -the ASR increases from 31% of the base agent to 36% of the reflexion agent with an attacked evaluator (Figure 5(A) and (D)).”

Even small perturbations to a single image in the environment can cause agents to pursue unintended goals.

“Even small perturbations to a single image in the environment can cause agents to pursue unintended goals.”

The captioner attack can make the captioner generate texts that are extremely out-of-distribution as a caption.

“We see that the attack can make the captioner generate texts that are extremely out-of-distribution as a caption.”

If the attacker attacks both the policy model and the evaluator, the evaluator more readily accepts adversarial actions (ASR on the evaluator to finish edge rises from 18% to 21%).

“Two key phenomena increase the ASR: (1) the evaluator more readily accepts adversarial actions (ASR on the evaluator to finish edge rises from 18% to 21%)”

The CLIP attack achieves an ASR of 19% on self-captioning agents.

“Figure 4(C) shows that the CLIP attack achieves an ASR of 19% on self-captioning agents.”

GPT-4o demonstrates the best robustness-utility trade-off with high Benign SR and low ASR.

“Among the different LMs, GPT-4o demonstrates the best robustness-utility trade-off with high Benign SR and low ASR.”

38% of the captions generated by the black-box LM captioner are adversarial (as seen by λ of that edge).

“We see that 38% of the captions generated by the black-box LM captioner are adversarial (as seen by λ of that edge).”

In the evaluations, 70% of webpages have more than 10 images.

“e.g., 70% of webpages in our evaluations have more than 10 images”

The ASR of the CLIP attack drops when not using self-caption.

“We see that the ASR of the CLIP attack drops when not using self-caption”

We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.

“we manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena”

• Also stated in: § 4.1 CURATION OF ADVERSARIAL TASKS · plain

  “We curated VWA-Adv, a set of 200 realistic adversarial tasks based on VWA.”

The CLIP attack has difficulty transferring when the image is embedded in a larger context (e.g., screenshot).

“suggesting that the attack has difficulty transferring when the image is embedded in a larger context (e.g., screenshot).”

Instruction hierarchy has not solved the problem.

“However, the absolute ASR on GPT-4o is still high, which means instruction hierarchy has not solved the problem.”

The explicit consistency check largely increases the number of API calls.

“However, this might not be desirable in practice since it largely increases the number of API calls”

The CLIP attack is robust to rescaling at test time.

“suggesting that this attack is robust to rescaling at test time.”

Methods that scale inference-time compute (e.g., reflexion and tree search) decrease robustness in worst-case scenarios.

“Methods that scale inference-time compute (e.g., reflexion and tree search) decrease robustness in worst-case scenarios.”

Deploying multimodal agents in real environments raises significant safety concerns because adversaries may attempt to manipulate the agent's actions through malicious inputs.

“The effectiveness of these attacks raises significant concerns about the safety of deploying multimodal agents in real environments, where adversaries may attempt to manipulate the agent's actions through malicious inputs.”

Inference-time compute that typically improves benign performance can open up new vulnerabilities and harm robustness.

“We find that inference-time compute that typically improves benign performance can open up new vulnerabilities and harm robustness.”

The overall ASR of the self-consistency check in the presence of CLIP attack is upper bounded by 38% against a determined adversary.

“The overall ASR of the self-consistency check in the presence of CLIP attack is therefore upper bounded by 38% against a determined adversary.”

Instruction hierarchy is helpful in the agent setting.

“This suggests the instruction hierarchy is helpful in the agent setting.”

• AssumptionThe authors state this as an assumption: GPT-4o is trained with instruction hierarchy, a training method that defends the LM from being distracted by untrusted inputs.

Adding uncompromised/robust new components can improve agent safety.

“Second, adding uncompromised/robust new components can improve agent safety.”

All components in an agent can be effectively attacked.

“First, all components in an agent can be effectively attacked.”

The work demonstrates adversarial attacks on multimodal agents, even in challenging scenarios with limited access to and knowledge about the agent's environment.

“Our work demonstrates the adversarial attacks on multimodal agents, even in challenging scenarios with limited access to and knowledge about the agent's environment.”