CoChat
Sign Up
PreprintCited by 7Adversarial Robustness in Machine LearningSecurity and Verification in ComputingPrivacy-Preserving Technologies in Data

Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks

Open access
Li, Ang, Zhou, Yin, Raghuram, Vethavikashini Chithrra, Goldstein, Tom, Goldblum, Micah·2025·arXiv
DOI 10.48550/arxiv.2502.08586arXiv 2502.08586
Open this paper in CoChat — free accountOpen PDF

Abstract

A high volume of recent ML security literature focuses on attacks against aligned large language models (LLMs). These attacks may extract private information or coerce the model into producing harmful outputs. In real-world deployments, LLMs are often part of a larger agentic pipeline including memory systems, retrieval, web access, and API calling. Such additional components introduce vulnerabilities that make these LLM-powered agents much easier to attack than isolated LLMs, yet relatively little work focuses on the security of LLM agents. In this paper, we analyze security and privacy vulnerabilities that are unique to LLM agents. We first provide a taxonomy of attacks categorized by threat actors, objectives, entry points, attacker observability, attack strategies, and inherent vulnerabilities of agent pipelines. We then conduct a series of illustrative attacks on popular open-source and commercial agents, demonstrating the immediate practical implications of their vulnerabilities. Notably, our attacks are trivial to implement and require no understanding of machine learning.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper argues that LLM-powered agents introduce security and privacy vulnerabilities beyond standalone LLMs, and demonstrates simple, non-ML attacks against commercial and open-source agents. It shows high success rates for leaking private data, downloading/executing files, sending authenticated phishing emails, and manipulating scientific agents toward toxic chemical synthesis.

Key findings

  • Web agents can be redirected from trusted platforms like Reddit to malicious sites and then leak private data: Anthropic’s Computer Use and MultiOn divulged sensitive information such as credit card numbers and addresses in 10 out of 10 trials when redirected via Reddit, while directly providing the malicious link failed in all 10 trials.
  • Anthropic’s Computer Use downloaded and executed a malicious file in 10 out of 10 trials when it landed on the attacker’s Reddit posts; MultiOn could not be extensively tested because its product went offline, but prior tests showed it reliably downloaded and opened an untrustworthy PDF.
  • Anthropic’s Computer Use was manipulated into sending phishing emails using the user’s credentials: two phishing messages were each tested 5 times, succeeding in all 10 attempts.
  • Scientific agents were vulnerable to database poisoning and obfuscation: PaperQA retrieved a fabricated malicious document in 100 out of 100 trials when queried for the best synthesis route, and ChemCrow reconstructed a step-by-step nerve gas procedure when queried using IUPAC nomenclature and indirect references.

Methods

  • The paper first provides a taxonomy of attacks on LLM agents by threat actors, objectives, entry points, attacker observability, attack strategies, and inherent agent-pipeline vulnerabilities.
  • It then conducts illustrative attacks using a four-step pipeline: agents access trusted platforms, encounter attacker posts, are redirected to malicious sites, and are jailbroken on those sites.
  • Evaluated agents include Anthropic’s Computer Use, MultiOn, ChemCrow, and PaperQA; the scientific-agent experiment used a synthetic database of over 10,000 chemical recipes with one inserted malicious document.

Limitations

  • The paper states that MultiOn underwent organizational changes and its product was disabled by the time success rates were recorded, preventing extensive success-rate analysis for some attacks.
  • The authors state they refrained from large-scale deployment of malicious posts and from attacking highly trafficked pages to avoid disrupting users and communities.
  • The text makes evident additional limitations: the tested agents are proprietary black boxes whose internal workings the authors do not know, trial counts are small for several web-agent attacks, and automated red-teaming/attack algorithms for agents are left as future work.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.08586

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 18 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Current LLM security research predominantly focuses on addressing vulnerabilities in isolated models, often concentrating on jailbreak attacks or extracting memorized training data.

    “Current research in LLM security predominantly focuses on addressing vulnerabilities in isolated models, often concentrating on jailbreak attacks (Zou et al., 2023;Liu et al., 2024) or on extracting memorized training data from a trained model (Nasr et al., 2023).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Zou et al., 2023 source
    • cites Liu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Nasr et al., 2023 source
  • Interpretationcitation

    LLM agents introduce new security challenges beyond the well-studied vulnerabilities of standalone LLMs.

    “Indeed, the security implications of LLM agents extend far beyond the well-studied vulnerabilities of standalone LLMs (Yang et al., 2024;Wang et al., 2024b;Ning et al., 2024;Tian et al., 2023).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Yang et al., 2024 source
    • cites Wang et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ning et al., 2024 source

      The Introduction (p. 2) presents a novel attack on LLM-empowered recommender systems, and the Conclusion (p. 8) reports their vulnerability to adversarial attacks. This supports a specific security challenge involving an LLM-based attack agent, but the paper does not establish the broader claim that LLM agents themselves introduce security challenges beyond those of standalone LLMs. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingnone given

    Existing LLM agents are susceptible to attacks that are simultaneously dangerous and trivial to implement by a user with no expertise related to machine learning.

    “We show that existing LLM agents are susceptible to attacks that are simultaneously dangerous and also trivial to implement by a user with no expertise related to machine learning.”
    Quote in the paperSubstantial claim
  • Findingown data

    Using Reddit posts, the authors convinced web agents to download and execute files from untrustworthy sources.

    “Similarly, we use Reddit posts to convince the web agents to download and execute files from untrustworthy sources.”
    Quote in the paperSubstantial claim
  • Findingnone given

    When the user's web browser is logged into an email client, the authors manipulated Anthropic's Computer Use web agent and MultiOn to send phishing emails to the user's contacts using the user's email credentials.

    “When the user's web browser is logged into an email client, we manipulate Anthropic's Computer Use web agent and MultiOn to send phishing emails to the user's contacts using the user's email credentials.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 31
  • May combine two claims. 22

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Other approaches may leverage reinforcement learning and few-shot red-teaming to systematically generate adversarial prompts.

    “Other approaches may leverage reinforcement learning (Deng et al., 2022;Wang et al., 2024a;Xue et al., 2024) and few-shot red-teaming (Perez et al., 2022) to systematically generate adversarial prompts.”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Deng et al., 2022 source

      The Introduction (p. 1) describes an RL policy that generates discrete prompts, supporting the reinforcement-learning part of the attribution. But §2.4 (p. 4) treats adversarial prompts as an unintended result of a vulnerable reward function and proposes a way to avoid them; it does not present systematic adversarial-prompt generation as the method's goal. The separate few-shot red-teaming citation is not attributed to this source. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Wang et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Xue et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationcitation

    Many cutting-edge attack algorithms involve gradient-based or transfer attacks, but these are impractical for LLM agents since they often require white-box access for computing gradients or a very large number of model calls.

    “Many cutting-edge attack algorithms involve gradient-based or transfer attacks (Zou et al., 2023), but these are impractical for LLM agents since they often require white-box access for computing gradients or require a very large number of model calls.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zou et al., 2023 source

  • May combine two claims.

    Interpretationcitation

    Attackers may introduce fake information into external databases or corrupt the agent's internal memory by injecting misleading information through repeated interactions.

    “For instance, attackers may introduce fake information into external databases (Goldblum et al., 2022;Chen et al., 2017;Schwarzschild et al., 2021;Tolpegin et al., 2020;Chen et al., 2024a) or corrupt the agent's internal memory by injecting misleading information through repeated interactions.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Goldblum et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chen et al., 2017 source

      The citation is attached to the external-databases clause. The paper describes attackers injecting poisoning samples into a model’s training dataset (B. Backdoor Adversary Using Data Poisoning, p. 4), which supports the general idea of introducing malicious data. It does not establish that attackers place fake information in external databases. The later internal-memory clause is not attributed to this citation. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Schwarzschild et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingreasoning

    The authors' attacks are trivial to implement and require no understanding of machine learning.

    “Notably, our attacks are trivial to implement and require no understanding of machine learning.”
    • Also stated in: § 1. Introduction · plain
      “We show that existing LLM agents are susceptible to attacks that are simultaneously dangerous and also trivial to implement by a user with no expertise related to machine learning.”
    • Also stated in: § 3. Breaking Commercial Web Agents · plain
      “Notably, our attacks require no understanding of machine learning.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Using Reddit posts to redirect agents to malicious web pages for fake products, the authors manipulated Anthropic's Computer Use web agent and MultiOn to leak private user data, including credit card numbers.

    “By using Reddit posts to redirect agents to malicious web pages for fake products, we manipulate Anthropic's Computer Use web agent and MultiOn to leak private user data, including credit card numbers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    ChemCrow, when asked for a synthesis procedure for the pharmaceutical compound Xadago, is manipulated into returning the recipe for nerve gas.

    “An example attack on ChemCrow.The agent is asked for a synthesis procedure for a pharmaceutical compound, Xadago, but is instead manipulated into returning the recipe for nerve gas.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The authors tested each phishing attack 5 times and succeeded in all 10 attempts.

    “We test each phishing attack 5 times and succeed in all 10 attempts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    LLM-powered agents, especially those that can communicate with the outside world via web access or external-facing databases, already pose a massive danger to their users which has largely been overlooked by the ML security and privacy community.

    “In this paper, we argue that LLM-powered agents, especially those that have the ability to communicate with the outside world via web access or external-facing databases, already pose a massive danger to their users which has largely been overlooked by the ML security and privacy community.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The information an attacker has about an agent determines their ability to craft effective attacks.

    “The information an attacker has about an agent determines their ability to craft effective attacks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    More sophisticated automated attack algorithms may be inapplicable to agents since they often require access to model outputs in order to provide signal to the optimizer.

    “More sophisticated automated attack algorithms may be inapplicable to agents since they often require access to model outputs in order to provide signal to the optimizer.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Downloading untrustworthy PDFs is known to carry a risk of viruses.

    “Downloading untrustworthy PDFs is known to carry a risk of virusesfoot_0 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites foot_0

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Jailbreak prompting bypasses model alignment or refusal messages and elicits harmful responses through handcrafted or optimized token sequences.

    “Jailbreak prompting (Liu et al., 2023;Shen et al., 2024;Wei et al., 2024), one of the most prevalent attacks on LLMs, bypasses model alignment or refusal messages and elicits harmful responses through handcrafted or optimized token sequences.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Liu et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Shen et al., 2024 source

      Not checked in the free run.

    • cites Wei et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    LLM agents are most vulnerable to attacks through their operational environment.

    “LLM agents are most vulnerable to attacks through their operational environment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Attackers can use the agent's access to external systems to impact further individuals and systems, for example by manipulating the agent to send phishing emails from the user's account to all his or her contacts.

    “In the second case, attackers can use the agent's access to external systems to impact further individuals and systems.For example, attackers may manipulate the agent to send phishing emails from the user's account to all his or her contacts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Malicious actors can manipulate the outputs of external tools to introduce erroneous data or unexpected behaviors by updating the API functions secretly.

    “Malicious actors can manipulate the outputs of these tools to introduce erroneous data or unexpected behaviors by updating the API functions secretly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Transfer attacks work best when the surrogate and target models behave similarly, yet in agentic settings, the agent may have many moving parts that are unknown to the attacker which might limit transferability.

    “Moreover, transfer attacks work best when the surrogate and target models behave similarly, yet in agentic settings, the agent may have many moving parts that are unknown to the attacker which might limit transferability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Scientific discovery agents such as ChemCrow and PaperQA introduce significant risks: if manipulated, they can be exploited to generate chemical synthesis protocols for lethal substances including nerve gas, potent toxins, and explosives.

    “However, these agents also introduce significant risks: if manipulated, they can be exploited to generate chemical synthesis protocols for lethal substances including nerve gas, potent toxins, and explosives.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Users are not privy to the contents of the long-term memory system and may be unaware that their credit card information is still vulnerable.

    “Users are not privy to the contents of the long-term memory system and may be unaware that their credit card information is still vulnerable.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Knowing that a web agent uses HTML or vision inputs could allow an attacker to craft specific multimodal adversarial attacks.

    “For example, knowing that a web agent uses HTML or vision inputs could allow an attacker to craft specific multimodal adversarial attacks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Web agents store email addresses, credit card details, and other sensitive user data, enabling greater autonomy but also exposing them to greater risks.

    “They also store email addresses, credit card details, and other sensitive user data, enabling greater autonomy but also exposing them to greater risks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Information about agent outputs can be used to reverse-engineer the agent's design and as a signal for optimizing attacks.

    “This information can be used to reverse-engineer the agent's design and as a signal for optimizing attacks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    In the threat landscape for LLM agents, models can be vulnerable to external attackers that operate outside the direct agent-user interaction loop by manipulating the agent's external dependencies, such as web content, data sources, or API responses, to induce unintended or harmful behaviors.

    “However, in the threat landscape for LLM agents, these models can be vulnerable to external attackers that operate outside the direct agentuser interaction loop by manipulating the agent's external dependencies, such as web content, data sources, or API responses, to induce unintended or harmful behaviors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    MultiOn is an ideal target for this attack due to its Chrome extension, which grants direct access to the user's email without requiring a login.

    “MultiOn is an ideal target for this attack due to its Chrome extension, which grants direct access to the user's email without requiring a login.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Attacks on agents are already a realistic threat and have the potential to severely degrade cybersecurity and information privacy infrastructure if left unchecked.

    “However, we argue that such attacks on agents are already a realistic threat and have the potential to severely degrade cybersecurity and information privacy infrastructure if left unchecked, as agents are increasingly incorporated into vast software.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Agents are increasingly incorporated into vast software.

  • May combine two claims.

    Premisecitation

    Standalone LLMs are vulnerable to jailbreak attacks within direct user-LLM interactions, while LLM agents face new risks due to their operational environment involving real-world interactions and their complex architecture incorporating memory systems and API calling.

    “While standalone LLMs are vulnerable to jailbreak attacks within direct user-LLM interactions, LLM agents face new risks due to their operational environment involving realworld interactions, and their complex architecture incorporating memory systems and API calling (Weng, 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Weng, 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Such techniques can evade detection while still achieving malicious objectives.

    “Such techniques can evade detection while still achieving malicious objectives.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    Many attacks can be mitigated by careful design of agents, but some attacks are difficult to detect because of their highly contextual nature.

    “While many attacks can be mitigated by careful design of agents, some attacks are difficult to detect because of their highly contextual nature.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Understanding the specific architecture of the agent, including the underlying LLM, the types of tools it uses, and its memory management system, allows attackers to tailor their attacks to exploit weaknesses in those components.

    “Understanding the specific architecture of the agent, including the underlying LLM, the types of tools it uses, and its memory management system, allows attackers to tailor their attacks to exploit weaknesses in those components.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Anyone with access to public databases or the most basic domain knowledge can embed harmful content into innocuous documents and manipulate agents to execute dangerous protocols.

    “Anyone with access to public databases or the most basic domain knowledge can embed harmful content into innocuous documents and manipulate agents to execute dangerous protocols.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    Unlike MultiOn, which uses a Chrome extension that allows pre-login to Gmail outside of the agent's workflow, Anthropic's agent does not allow for pre-login.

    “Unlike MultiOn, which uses a Chrome extension that allows pre-login to Gmail outside of the agent's workflow, Anthropic's agent does not allow for pre-login.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    If an attacker who places a malicious document into a database can observe the hit rate at which users retrieve that document, they can optimize the document to be retrieved at a higher frequency.

    “For example, if an attacker who places a malicious document into a database can observe the hit rate at which users retrieve that document, they can then optimize the document to be retrieved at a higher frequency.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    An attacker targeting a web agent can create adversarial web content, including phishing websites, jailbreak text, or adversarial images, to inject harmful instructions into a web agent's decision-making process.

    “For instance, an attacker targeting a web agent can create adversarial web content, including phishing websites, jailbreak text, or adversarial images, to inject harmful instructions into a web agent's decision-making process.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Agents like MultiOn allow a user to provide credit card information to the agent, but this private information may persist in the agent's long-term memory in future interactions long after shopping.

    “Agents like Mul-tiOn allow a user to provide credit card information to the agent, but this private information may persist in the agent's long-term memory in future interactions long after shopping.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Due to the perceived credibility of these repositories, such documents can bypass the agent's safeguards and be retrieved by the agent's API tools.

    “Due to the perceived credibility of these repositories, such documents can bypass the agent's safeguards and be retrieved by the agent's API tools.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Scientific agents often rely on public resources such as PubMed, bioRxiv, arXiv, or other trusted repositories for data retrieval.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Attackers can trick agents into performing harmful actions that negatively impact the user, such as downloading malware on the user's local computer or executing unauthorized financial transactions.

    “In the first case, attackers can trick agents into performing harmful actions that negatively impact the user, such as downloading malware on the user's local computer or executing unauthorized financial transactions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The safeguards in scientific discovery agents like ChemCrow are inadequate and can be bypassed easily through obfuscation and indirect references.

    “While this approach can filter out some malicious content, it is far from sufficient, as these mechanisms are easily circumvented through obfuscation and indirect references.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The safeguards implemented in scientific discovery agents like ChemCrow are designed to identify and block the use of controlled chemicals, explosives, and other hazardous materials.
    • Rests on an assumption: These safeguards typically block specific names or direct references to dangerous substances within the input queries or retrieved data.
    • Rests on an assumption: ChemCrow and similar agents use a rule-based or heuristic approach to detect harmful content.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The consequences are severe, as even a well-intentioned user could be misled into producing and distributing deadly chemicals while under the impression that the chemicals are safe.

    “The consequences are severe, as even a well-intentioned user could be misled into producing and distributing deadly chemicals while under the impression that the chemicals are safe.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Attackers may gain visibility into the agent's outputs such as log files, actions it has taken, API responses, and LLM-generated text.

    “Attackers may gain visibility into the agent's outputs such as log files, actions it has taken, API responses, and LLM-generated text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    At the time the authors tested these attacks, MultiOn's service was disabled.

    “However, at the time we tested these attacks, MultiOn's service was disabled.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The ability of LLM agents to take actions in the real world without supervision extends the range of harmful practical consequences of attacks.

    “The ability of LLM agents to take actions in the real world without supervision extends the range of harmful practical consequences of attacks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Privacy leakages may especially occur when agents using RAG obtain information from a knowledge base containing private documents such as emails to generate responses.

    “Privacy leakages may especially occur when agents using RAG obtain information from a knowledge base, containing private documents such as emails, to generate responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    External attackers represent a new and significant threat, but malicious users still remain a concern for LLM agents.

    “While external attackers represent a new and significant threat, malicious users still remain a concern for LLM agents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Redirection to a malicious site avoids directly placing harmful instructions on the trusted platform, preventing detection and banning of the attacker's posts while enhancing the malicious site's credibility and increasing the likelihood of bypassing the agent's security checks.

    “This redirection avoids directly placing harmful instructions on the trusted platform, preventing detection and banning of the attacker's posts while enhancing the malicious site's credibility and increasing the likelihood of bypassing the agent's security checks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Attackers may manipulate agents to disrupt a user's local system or to disrupt broader external environments.

    “Attackers may either manipulate agents to disrupt a user's local system or to disrupt broader external environments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    By deploying a large volume of adversarial posts, attackers can ensure that LLM agents frequently encounter them.

    “By deploying a large volume of these adversarial posts, attackers can ensure that LLM agents frequently encounter them.”
    Quote in the paperNot a substantial claim
45 of 45

  • The authors state this as an assumption.

    User credentials reside in the system prompt for Anthropic's Computer Use.

    “Instead, we assume that user credentials reside in the system prompt.”

  • The paper relies on this without support.

    Many jailbreak attacks in practice elicit information readily available through standard search engines.

    “While valuable for academic purposes and potentially dangerous when applied to powerful models in the future, many of these jailbreak attacks in practice elicit information readily available through standard search engines.”

  • The paper relies on this without support.

    Scientific discovery agents rely heavily on external databases and retrieval-augmented generation (RAG) mechanisms to source information and provide contextually relevant responses.

    “Scientific discovery agents rely heavily on external databases and retrieval-augmented generation (RAG) mechanisms to source information and provide contextually relevant responses.”

  • The paper relies on this without support.

    Many scientific discovery agents include mechanisms to detect controlled chemicals, explosives, or other harmful content.

    “Many scientific discovery agents include mechanisms to detect controlled chemicals, explosives, or other harmful content.”

  • The paper relies on this without support.

    The safeguards implemented in scientific discovery agents like ChemCrow are designed to identify and block the use of controlled chemicals, explosives, and other hazardous materials.

    “The safeguards implemented in scientific discovery agents like ChemCrow are designed to identify and block the use of controlled chemicals, explosives, and other hazardous materials.”

  • The paper relies on this without support.

    These safeguards typically block specific names or direct references to dangerous substances within the input queries or retrieved data.

    “These safeguards typically block specific names or direct references to dangerous substances within the input queries or retrieved data.”

  • The paper relies on this without support.

    ChemCrow and similar agents use a rule-based or heuristic approach to detect harmful content.

    “ChemCrow and similar agents use a rule-based or heuristic approach to detect harmful content.”

  • The paper relies on this without support.

    The IUPAC name inherently includes detailed information about the compound's molecular structure and, in some cases, hints at its synthesis route.

    “This vulnerability is particularly alarming because the IUPAC name inherently includes detailed information about the compound's molecular structure and, in some cases, hints at its synthesis route.”

  • The paper relies on this without support.

    Agents are increasingly incorporated into vast software.

    “as agents are increasingly incorporated into vast software.”

  • The paper relies on this without support.

    Scientific agents often rely on public resources such as PubMed, bioRxiv, arXiv, or other trusted repositories for data retrieval.

    “Scientific agents often rely on public resources such as PubMed, bioRxiv, arXiv, or other trusted repositories for data retrieval.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • MultiOn phishing success is claimed despite no phishing tests on MultiOn

    Major

    The paper claims it manipulated both agents into sending phishing emails, but the phishing experiment section says MultiOn was disabled and experiments were performed on Anthropic's Computer Use instead.

  • Phishing setup claims a logged-in browser for an agent that does not allow pre-login

    Minor

    The overview describes the phishing attack as occurring when the user's browser is logged into email, including for Anthropic's agent. The experiment section instead says Anthropic's agent does not allow pre-login and assumes the credentials are in the system prompt.

2 of 2

Cites

Flagged first, then by citations

2 of the 45 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases, 2024b. URL https://arxiv.org/abs/2407.12784. · arXiv

Cited by

Sorted by citations

Related

Authors
Li, A., Zhou, Y., Raghuram, V. C., Goldstein, T., & Goldblum, M.
Affiliations
  • Columbia University (US)
  • University of Maryland, College Park (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.08586
arXiv
2502.08586
OpenAlex
W4417315191
Cited by
2
References
45
Citations by year
2025: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
Keywords
LLM agents, large language model security, privacy vulnerabilities, attack taxonomy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review