Attacking Vision-Language Computer Agents via Pop-ups

Prompting the model to ignore the pop-ups almost provides no mitigation effects.

“Surprisingly, prompting the model to ignore the pop-ups almost provides no mitigation effects.”

Some successful attack examples do not mention any elements from the pop-ups but generate actions that implicitly follow the instructions.

“More interestingly, we find some successful examples without mentioning any elements from the pop-ups but generating actions that implicitly follow the instructions (Figure 6).”

Basic defense techniques, such as asking the agent to ignore pop-ups or including an advertisement notice, are ineffective against the attack.

“Basic defense techniques, such as asking the agent to ignore pop-ups or including an advertisement notice, are ineffective against the attack.”

• Also stated in: § 1 Introduction · plain

  “Basic defense strategies, such as asking the agent to ignore pop-ups in system prompts and adding an extra advertisement notice, cannot effectively mitigate the issue (decrease the ASR by no more than 25% relatively).”

More than 50% of tested VisualWebArena tasks can be completed within five steps.

“we find that more than 50% of the tested VisualWebArena tasks can be completed within five steps”

TASR is usually multiple times higher than ASR when ASR is small (<0.2).

“More surprisingly, TASR is usually multiple times higher than ASR when ASR is small (< 0.2)”

In OSWorld, all VLM agents achieve SR ≤10% with the default attack even on the selected easy set.

“In OSWorld, it is hard for all VLM agents to achieve any meaningful SR with our default attack (≤ 10%) even with our selected easy set”

A more specific defensive prompt targeting clicking instructions leads to a reasonable but not satisfying decrease in ASR (relatively 25%) while SRs remain low.

“This detailed prompt defense leads to a reasonable but not satisfying decrease in ASR (relatively 25%) while the SRs remain low.”

Decreasing the size of pop-ups by 50% leads to a small decrease in ASR.

“we decrease the size of pop-ups by 50% and find it leads to a small decrease in ASR.”

Our attack achieves an attack success rate over 80% on OSWorld in the default setting.

“we find that our attack achieves an attack success rate (ASR) over 80% on OSworld and over 60% on Visual-WebArena in the default setting”

• AssumptionThe authors state this as an assumption: In the default setting, the attacker has complete information, including the user query, the pop-up's position, and the underlying agent framework.

No model shows exceptional robustness toward our attack.

“No model shows exceptional robustness toward our attack.”

More than 50% of OSWorld tasks only stop after reaching the 15-step limit.

“(more than 50% of tasks only stop after reaching the 15-step limit).”

On average, 92.7% of all actions of attacked agents in OSWorld are clicking on the adversarial pop-ups.

“On average, 92.7% / 73.1% of all actions of attacked agents in OSWorld/VisualWebArena are clicking on the adversarial pop-ups.”

TASR is generally positively associated with ASR.

“we find that TASR is generally positively associated with ASR”

All tested VLM agents backed by different models exhibit high ASR (>60%) in all scenarios.

“All models exhibit high ASR (> 60%) in all scenarios, demonstrating the lack of safety awareness related to pop-ups.”

In VisualWebArena, all SRs remain around 45% after being attacked.

“while all SRs remain around 45% after being attacked in VisualWebArena.”

Using click (x,y) instead of click [ID] to attack SoM agents causes only a slight drop of ASR (-4.3% and -12.8%).

“Assuming we have no knowledge of the attacked agent framework and simply use”

• AssumptionRests on an assumption: The tag ID is usually generated from the agent framework, where attackers have zero knowledge.

On average, 73.1% of all actions of attacked agents in VisualWebArena are clicking on the adversarial pop-ups.

“On average, 92.7% / 73.1% of all actions of attacked agents in OSWorld/VisualWebArena are clicking on the adversarial pop-ups.”

In OSWorld, the attacked agent can easily get stuck in the middle and cannot complete the task within the limit in most cases (≥80%).

“In this case, the attacked agent can easily get stuck in the middle and cannot complete the task within the limit in most cases (≥ 80%).”

Agents typically remain effective when there are more confident and certain actionable elements than the current pop-ups in the observation.

“In addition to these scenarios, agents typically remain effective when there are more confident and certain actionable elements than the current pop-ups in the observation.”

VLM agents can be easily maliciously instructed by adversarial popups while operating on computers.

“In this work, we show that VLM agents can be easily maliciously instructed by adversarial popups while operating on computers.”

A simple attack variant by removing the phrase Please click before the coordinates in the malicious instructions can effectively increase the ASR without any search or optimization.

“A simple attack variant by removing the phrase”

When the answer is directly provided elsewhere on the current page, forcing the agent to click on pop-ups instead of returning answers is hard.

“When such an answer is directly provided elsewhere on the current page, forcing the agent to click on pop-ups instead of returning answers is hard.”

With the attack, agent thoughts become more specific, usually mentioning elements from the pop-ups, such as target coordinates, tags, the attention hook, and the OK in the info banner.

“With the attack, the thoughts become more specific, usually mentioning the elements from the pop-ups, such as the target coordinates (example 1 and 2) and tags (example 3), the attention hook (example 2), and the”

Our attack achieves an attack success rate over 60% on VisualWebArena in the default setting.

“we find that our attack achieves an attack success rate (ASR) over 80% on OSworld and over 60% on Visual-WebArena in the default setting”

• AssumptionThe authors state this as an assumption: In the default setting, the attacker has complete information, including the user query, the pop-up's position, and the underlying agent framework.

Other information such as position and agent framework information is relatively unnecessary to make the attack successful.

“(2) Other information (e.g., position and agent framework information) is relatively unnecessary to make the attack successful.”

Even with speculated user queries, the attack still shows considerable ASRs (33.3% on average).

“Even with speculated user queries, our attack still shows considerable ASRs (33.3% on average)”

The attack fails when agents declare WAIT/FAIL/DONE based on the interaction history.

“(i) Agents declare WAIT/FAIL/DONE based on the interaction history.”

Even with the attack, VLM agents complete fewer but still considerable VisualWebArena tasks within five steps.

“Even with our attack, VLM agents complete fewer but still considerable tasks within five steps.”

Changing the summarized query to a virus alert causes a more dramatic drop of ASR in SoM agents (-33.5% and -61.0%) compared to screenshot agents (-3.3%).

“by changing the summarized query to virus alert, we observe a more dramatic drop of ASR (-33.5% and -61.0%) in SoM agents compared to screenshot agents (-3.3%).”

Agents can be trapped by constantly clicking popups without effectively exploring the environment.

“Agents can be trapped by constantly clicking popups without effectively exploring the environment.”

The high ASR demonstrates a lack of safety awareness related to pop-ups in VLM agents.

“All models exhibit high ASR (> 60%) in all scenarios, demonstrating the lack of safety awareness related to pop-ups.”

The study only tests the performance of closed-source models.

“(i) We only test the performance of closed-source models, which hinders a deeper understanding of why such an attack works.”

The attack has potential to be successful without access to the user's intents.

“suggesting the potential of a successful attack without access to the user's intents.”

• AssumptionRests on an assumption: The ground truth user query is tricky to access from the attacker perspective in real scenarios.

VLM agents can be easily attacked by carefully designed adversarial pop-ups that human users would typically recognize and ignore.

“In this work, we demonstrate that VLM agents can be easily attacked by a set of carefully designed adversarial pop-ups 1 , which human users would typically recognize and ignore.”

The behavior of generating actions that implicitly follow instructions without mentioning pop-up elements increases the stealthiness of the attack.

“Considering one potential defense strategy is to check whether the generated thoughts follow suspicious instructions, this kind of behavior increases the stealthiness of the attack.”