AdvAgent: Controllable Blackbox Red-teaming on Web Agents

May combine two claims.

In black-box settings, the attacker has no access to model parameters or gradients, relying solely on agent responses, which makes gradient-based optimization techniques ineffective.

“The attacker has no access to the model's parameters or gradients, relying solely on the agent's responses to adversarial inputs, which makes gradient-based optimization techniques (Zou et al., 2023) ineffective.”

• cites Zou et al., 2023 sourceDisputed

  Section 2.2 describes Zou et al.’s gradient-based optimization method, and the Introduction reports that its prompts successfully attacked GPT-3.5 and GPT-4 through transfer. The claim that gradient-based techniques are ineffective in black-box settings overlooks this distinction: the paper uses gradients from accessible models to generate attacks that transfer to proprietary targets. (LLM, conf=0.86)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: The attacker operates in a black-box setting with no access to the agent framework, backend model weights, or logits.

May combine two claims.

SeeAct is a state-of-the-art web agent powered by different proprietary VLMs.

“To demonstrate the effectiveness of AdvAgent, we employ SeeAct (Zheng et al., 2024), a state-of-the-art web agent powered by different proprietary VLMs (Achiam et al., 2023;Team et al., 2023).”

• cites Zheng et al., 2024 sourceSupported
• cites Achiam et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Team et al., 2023 sourcePartly supported

  Section 2 (Model Architecture; page unavailable) describes Gemini models as accepting visual and textual inputs, supporting their identification as multimodal models. The paper does not say that SeeAct uses Gemini or establish which models power SeeAct. The separate Zheng et al. citation, rather than this source, is attached to SeeAct’s identification. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

AdvAgent is specifically designed for black-box web agent attacks, incorporating model feedback for superior performance.

“Unlike existing LLM red-teaming approaches (Deng et al., 2023; Ge et al., 2023; Paulus et al., 2024), AdvAgent is specifically designed for black-box web agent attacks, incorporating model feedback for superior performance.”

• cites Deng et al., 2023 sourceSupported
• cites Ge et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Paulus et al., 2024 sourceSupported

• AssumptionThe authors state this as an assumption: The attacker operates in a black-box setting with no access to the agent framework, backend model weights, or logits.

May combine two claims.

Yang et al. (2024) and Wang et al. (2024) explore backdoor attacks by inserting triggers into web agents through fine-tuning backbone models with white-box access, misleading agents into making incorrect decisions.

“Yang et al. (2024) and Wang et al. (2024) explore backdoor attacks by inserting triggers into web agents through fine-tuning backbone models with white-box access, misleading agents into making incorrect decisions.”

• cites Yang et al., 2024 sourceSupported
• cites Wang et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Table 2 shows mean ASR for different position is 71.0 ± 26.1 and for different HTML field is 97.0 ± 1.7.

“Different Position 26.0 82.0 88.0 88.0 71.0 ± 26.1 Different HTML Field 98.0 94.0 98.0 98.0 97.0 ± 1.7”

May combine two claims.

AdvAgent allows attackers to easily control and modify generated successful injection prompts without requiring re-optimization, making it easy to achieve different attack goals for the same user query.

“Besides, AdvAgent allows attackers to easily control and modify generated successful injection prompts without requiring re-optimization, making it easy to achieve different attack goals for the same user query, such as targeting different companies or actions, with minimal additional effort.”

• AssumptionThe authors state this as an assumption: The attack should be easily adaptable to new targets without additional interaction or re-optimization with the agent.

May combine two claims.

Transferring successful injection prompts between different models yields limited ASR, emphasizing the importance of black-box red-teaming over existing transfer-based approaches.

“Finally, we highlight that transferring successful injection prompts between different models yields limited ASR, emphasizing the importance of our black-box red-teaming algorithm over existing transfer-based approaches.”

May combine two claims.

AdvAgent's prompts generalize to new targets through a simple replacement function D, exposing significant vulnerabilities in real-world web agent deployments.

“Our findings reveal that the prompts generated by AdvAgent is able to generalize to new targets through a simple replacement function D, exposing significant vulnerabilities in real-world web agent deployments.”

• AssumptionThe authors state this as an assumption: The attack should be easily adaptable to new targets without additional interaction or re-optimization with the agent.

May combine two claims.

Ablation studies show the proposed two-stage training framework is crucial, and learning from model feedback significantly enhances the effectiveness of the attack.

“Furthermore, we conduct ablation studies, showing that the proposed two-stage training framework is crucial, and learning from model feedback significantly enhances the effectiveness of the attack.”

May combine two claims.

AdvAgent maintains high adaptability and remains effective even against defense strategies, achieving an ASR above 88.8%.

“Further analysis reveals that AdvAgent maintains high adaptability and remains effective even against defense strategies, achieving an ASR above 88.8%.”

May combine two claims.

AdvAgent achieves a 97.5% attack success rate against GPT-4V-based SeeAct across different website domains, significantly outperforming baseline methods.

“Our results demonstrate that AdvAgent is highly effective, achieving a 97.5% attack success rate (ASR) against GPT-4V-based SeeAct across different website domains, significantly outperforming baseline methods.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Subtle differences in adversarial prompts lead to different attack results.

“Subtle differences in adversarial prompts lead to different attack results.”

May combine two claims.

ASR decreases in the Finance domain but remains high in others, with near-perfect attack success rates even after applying defenses.

“Notably, while ASR decreases in the Finance domain, it remains high in others, with near-perfect attack success rates even after applying defenses.”

May combine two claims.

Common defense strategies reduce ASR in some cases, but their effectiveness varies across different domains.

“While these defenses reduce ASR in some cases, their effectiveness varies across different domains.”

May combine two claims.

Transfer-based attacks achieve a low ASR of only 18.0%, demonstrating poor generalization across different backend models.

“As shown in Table 4, transfer-based attacks achieve a low ASR of only 18.0%, demonstrating poor generalization across different backend models.”

May combine two claims.

Transfer-based attacks exhibit low ASR, as successful attacks on one model do not transfer well to another.

“Transferbased attacks exhibit low ASR, as successful attacks on one model do not transfer well to another.”

May combine two claims.

Modifying the HTML field has minimal impact, with ASRs remaining consistently high across all domains and an average ASR of 97.0%.

“In contrast, modifying the HTML field has minimal impact, where ASRs remain consistently high across all domains, with an average ASR of 97.0%.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A malicious website developer could exploit routine maintenance or updates to inject adversarial modifications, compromising user safety for financial gain.

“For example, a malicious website developer could exploit routine maintenance or updates to inject adversarial modifications, compromising user safety for financial gain.”

Probably not a substantial claim: may not pass S4 (based).

The decision space of adversarial HTML content is discrete, making optimization inherently difficult.

“The decision space of adversarial HTML content h adv is discrete, making optimization inherently difficult.”

May combine two claims.

By employing DPO, AdvAgent learns from both successful and unsuccessful attack attempts against the black-box web agent, enabling flexible and efficient attacks.

“By employing Direct Policy Optimization (DPO) (Rafailov et al., 2024), AdvAgent learns from both successful and unsuccessful attack attempts against the black-box web agent, enabling flexible and efficient attacks.”

• cites Rafailov et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The attacker operates in a black-box setting with no access to the agent framework, backend model weights, or logits.

Classified as probably not a substantial claim.

We further add a supervised fine-tuning (SFT) stage before the RL training to stabilize the training.

“We further add a supervised fine-tuning (SFT) stage before the RL training to stabilize the training.”

• AssumptionRests on an assumption: Reinforcement learning is unstable in the training process.

May combine two claims.

AdvAgent trains a generative model to automate adversarial string generation, improving efficiency and scalability while minimizing manual effort.

“Third, AdvAgent trains a generative model to automate adversarial string generation, improving efficiency and scalability while minimizing manual effort.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Mind2Web includes 2,350 tasks from 137 websites across 31 domains.

“This dataset includes 2, 350 tasks from 137 websites across 31 domains.”

• cites Deng et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Transfer-based red-teaming methods suffer from limited success rates due to backend differences.

“Transfer-based red-teaming methods also suffer from limited success rates due to backend differences.”

May combine two claims.

By leveraging an RL-based algorithm, AdvAgent efficiently optimizes adversarial prompt generation, adapting to nuanced attack patterns.

“By leveraging an RL-based algorithm, it efficiently optimizes adversarial prompt generation, adapting to nuanced attack patterns.”

May combine two claims.

Wu et al. (2024a) introduces automatic adversarial input optimization for misleading web agents, but their approach is either impractical, requiring white-box access for gradient-based optimization, or achieves limited success rate when transferring attacks across multiple CLIP models to blackbox agents.

“Wu et al. (2024a) introduces automatic adversarial input optimization for misleading web agents, but their approach is either impractical, requiring white-box access for gradientbased optimization, or achieves limited success rate when transferring attacks across multiple CLIP models to blackbox agents.”

• cites Wu et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

To maintain stealthiness, q is placed within non-rendered HTML attributes (e.g., aria-label) so that it does not alter the website's appearance.

“To maintain stealthiness, q is placed within non-rendered HTML attributes (e.g., aria-label) so that it does not alter the website's appearance.”

May combine two claims.

AdvAgent works in black-box settings, without access to the agent weights or logits.

“Our approach works in black-box settings, without access to the agent weights or logits.”

• AssumptionThe authors state this as an assumption: The attacker operates in a black-box setting with no access to the agent framework, backend model weights, or logits.

Probably not a substantial claim: may not pass S3 (load-bearing).

Benign developers may unknowingly introduce vulnerabilities by integrating contaminated libraries.

“Additionally, benign developers may unknowingly introduce vulnerabilities by integrating contaminated libraries, as highlighted in a recent CISA report (Synopsys, 2024), where the resulting websites may contain hidden but exploitable vulnerabilities.”

• cites Synopsys, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The malicious instructions in prior works are manually designed and written with heuristics, leading to limited scalability and flexibility.

“However, the malicious instructions are manually designed and written with heuristics (Wu et al., 2024c;b), leading to limited scalability and flexibility.”

• cites Wu et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wu et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

SeeAct operates by first generating an action description based on the current task and the webpage screenshot, then mapping this description to the corresponding HTML contents to interact with the web environment.

“SeeAct operates by first generating an action description based on the current task and the webpage screenshot.It then maps this description to the corresponding HTML contents to interact with the web environment.”

• cites Zheng et al., 2024 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Adversarial prompts are injected at the location corresponding to the agent's expected element choice e, aligning with the agent's action prediction mechanism.

“Additionally, we inject adversarial prompts at the location corresponding to the agent's expected element choice e, aligning with the agent's action prediction mechanism.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The injected string is hidden in invisible HTML fields and does not change the website rendering.

“The injected string is hidden in invisible HTML fields and does not change the website rendering.”

Classified as probably not a substantial claim.

To enforce controllability, placeholders (e.g., {target argument}) are embedded in q, enabling easy substitution of attack targets without reoptimization.

“To enforce controllability, we embed placeholders (e.g., {target argument}) in q, enabling easy substitution of attack targets without reoptimization.”

• AssumptionThe authors state this as an assumption: The attack should be easily adaptable to new targets without additional interaction or re-optimization with the agent.

May combine two claims.

AdvAgent integrates black-box agent feedback and trains a prompter model, allowing automated and adaptive prompt generation, leading to superior performance.

“Unlike InjecAgent, AdvAgent integrates black-box agent feedback and trains a prompter model, allowing for automated and adaptive prompt generation, leading to superior performance.”

Classified as probably not a substantial claim.

Step-level evaluation does not fully capture the overall risks associated with web agents in completing entire user requests.

“While this step-level evaluation provides valuable insights, it does not fully capture the overall risks associated with web agents in completing entire user requests.”

Classified as probably not a substantial claim.

These findings highlight security risks in sensitive domains such as finance, healthcare, and data security.

“These findings highlight security risks in sensitive domains such as finance, healthcare, and data security”

May combine two claims.

The feedback-driven black-box attack strategy is effective and superior to traditional transfer-based approaches.

“These results underscore the effectiveness of our feedbackdriven black-box attack strategy and highlight its superiority over traditional transfer-based approaches.”

May combine two claims.

Injection position plays a crucial role in attack effectiveness and may require task-specific tuning.

“This suggests that injection position plays a crucial role in attack effectiveness and may require task-specific tuning.”

Classified as probably not a substantial claim.

In this work, obtaining feedback from the victim agent before performing attack string optimization must be done offline.

“In this work, we require obtaining feedback from the victim agent before performing attack string optimization, which must be done offline.”

Classified as probably not a substantial claim.

The evaluation focuses on the step-based attack success rate (ASR), assessing the success of adversarial attacks at individual action steps.

“Additionally, our evaluation focuses on the step-based attack success rate (ASR), where we assess the success of adversarial attacks at individual action steps.”

• AssumptionRests on an assumption: Current web agents have relatively low end-to-end task completion rates.