CoChat
Sign Up
PreprintCited by 2Adversarial Robustness in Machine LearningSecurity and Verification in ComputingWeb Application Security Vulnerabilities

Multi-Agent Systems Execute Arbitrary Malicious Code

Open access
Triedman, Harold, Jha, Rishi, Shmatikov, Vitaly·2025·arXiv
DOI 10.48550/arxiv.2503.12188arXiv 2503.12188
Open this paper in CoChat — free accountOpen PDF

Abstract

Multi-agent systems coordinate LLM-based agents to perform tasks on users' behalf. In real-world applications, multi-agent systems will inevitably interact with untrusted inputs, such as malicious Web content, files, email attachments, and more. Using several recently proposed multi-agent frameworks as concrete examples, we demonstrate that adversarial content can hijack control and communication within the system to invoke unsafe agents and functionalities. This results in a complete security breach, up to execution of arbitrary malicious code on the user's device or exfiltration of sensitive data from the user's containerized environment. For example, when agents are instantiated with GPT-4o, Web-based attacks successfully cause the multi-agent system execute arbitrary malicious code in 58-90\% of trials (depending on the orchestrator). In some model-orchestrator configurations, the attack success rate is 100\%. We also demonstrate that these attacks succeed even if individual agents are not susceptible to direct or indirect prompt injection, and even if they refuse to perform harmful actions. We hope that these results will motivate development of trust and security models for multi-agent systems before they are widely deployed.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces multi-agent system control-flow hijacking (MAS hijacking), where adversarial content manipulates inter-agent metadata and error handling to make LLM-based multi-agent systems execute arbitrary malicious code or exfiltrate user data. Across AutoGen, CrewAI, and MetaGPT, attacks succeed at high rates even when individual agents refuse harmful actions and when prior indirect prompt injection attacks fail.

Key findings

  • Web-based MAS hijacking attacks against GPT-4o-instantiated systems achieved arbitrary malicious code execution in 58–90% of trials depending on the orchestrator, with some model-orchestrator configurations reaching 100% attack success.
  • Specific reported rates include Magentic-One with GPT-4o executing malicious local files 97% of the time, Magentic-One with Gemini 1.5 Pro executing malicious web pages 88% of the time, and CrewAI with GPT-4o exfiltrating private user data from a local file with 65% success.
  • Previously known indirect prompt injection (IPI) baselines were almost completely unsuccessful, with IPI ASR of 0–1% across most tested orchestrators, while direct asks to open reverse shells were dominated by Gemini models (61 of 67 successful direct-ask executions).
  • Additional experiments found 87.5% ASR (35/40) under incidental contact, all 10 paraphrased user queries succeeded with ASR ranging from 40% to 85% and average 59.5% across 200 trials, and direct submission of adversarial metadata to the orchestrator achieved 80% ASR (28/40) with 0% refusal.

Methods

  • The authors formalized MAS hijacking as adversarial manipulation of shared metadata and inter-agent communication, then tested it against AutoGen (Magentic-One, Selector, Round-Robin orchestrators), CrewAI (default orchestrator), and MetaGPT (Data Interpreter), using GPT-4o, GPT-4o-mini, Gemini 1.5 Pro, and Gemini 1.5 Flash.
  • Attacks used local text files, web text, web images, and local videos; three error templates (fake Python, WordPress 403, generic “Access Denied”); two user-prompt wordings; and reverse-shell payloads in non-obfuscated Python, with CrewAI also tested for data exfiltration. Each framework-orchestrator-model-input-error-query tuple was run 10 times.
  • Attack success was measured by modifying the code execution agent to exit and write a unique string when it would run the payload, backed by a cloud server monitoring reverse-shell connections; baselines included benign tasks, direct malicious requests, and IPI templates from Debenedetti et al. (2024). Additional experiments used Magentic-One with GPT-4o-mini: incidental contact (40 trials), prompt paraphrases (20 trials per paraphrase), and direct hijack (40 trials).

Limitations

  • The paper states its investigation is limited to three prominent open-source frameworks, excludes simulated or purely conversational multi-agent systems and highly customizable business/developer-focused frameworks, and could only evaluate CrewAI with OpenAI models due to a known bug affecting agent tool usage.
  • The paper states that the underlying LLMs are black-box and training details are unavailable, that all experiments were run in a controlled lab environment with no production services attacked, and that video/audio attack results were inconsistent.
  • Evident from the text: the evaluation covers a limited set of models and orchestrators, uses 10 trials per main tuple, does not evaluate proposed defenses against MAS hijacking, and focuses on two malicious goals (reverse shell and data exfiltration), so broader generalization and defense effectiveness remain untested.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.12188

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 7 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisenone given
    § 7.1 AI agent security · p. 9

    No prior work considered attacks on MAS metadata and control flow processes.

    “No prior work considered attacks on MAS metadata and control flow processes.”
    • Also stated in: § A.2 Attacks on multi-agent systems · p. 17 · plain
      “None of the prior work considered attacks on MAS metadata and control flows.”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Lee & Tiwari (2024) source

      The Introduction says most MAS-safety studies focus on errors or noise in agent behavior, and the Safety in Multi-Agent Systems section discusses prior prompt-injection attacks. Neither passage addresses whether prior work considered attacks on MAS metadata or control-flow processes. Searches for those terms found no passage making that claim. (LLM, conf=0.74)

      Experimental — accuracy not yet measured
    • cites Cohen et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ju et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data
    § Abstract · p. 1

    When agents are instantiated with GPT-4o, Web-based MAS hijacking attacks cause the multi-agent system to execute arbitrary malicious code in 58-90% of trials, depending on the orchestrator.

    “For example, when agents are instantiated with GPT-4o, Web-based attacks successfully cause the multi-agent system execute arbitrary malicious code in 58-90% of trials (depending on the orchestrator).”
    Quote in the paperSubstantial claim
  • Findingown data
    § 6.3 Varying attack payload · p. 8

    Success rates of data exfiltration attacks depend on the quality of the orchestrator model.

    “Success rates also depend on the quality of the orchestrator model.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Abstract · p. 1

    In some model-orchestrator configurations, the MAS hijacking attack success rate is 100%.

    “In some model-orchestrator configurations, the attack success rate is 100%.”
    • Also stated in: § 1 Introduction · p. 2 · plain
      “For certain model-orchestrator combinations, the attack success rate is 100%.”
    Quote in the paperNot a substantial claim
  • Findingnone given
    § 6.1 MAShijacking works, indirect prompt injection doesn't · p. 7

    GPT models typically refuse requests to create and execute malicious code.

    “GPT models, on the other hand, typically refuse requests to create and execute malicious code.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 32
  • May combine two claims. 31
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 5 Experimental setup · p. 6

    The experiments were compared against baseline tasks (e.g., benign file access), direct malicious requests, and previously studied IPI templates from the Debenedetti et al. (2024) Github repository.

    “Our experiments were compared against baseline tasks (e.g., benign file access), direct malicious requests, and previously studied IPI templates from the Debenedetti et al. (2024) Github repository.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Debenedetti et al. (2024) source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 7.1 AI agent security · p. 9

    Indirect prompt injection (IPI) is an attack vector introduced by Greshake et al. (2023) and systematized by Zhan et al. (2024).

    “Indirect prompt injection (IPI) is an attack vector introduced by Greshake et al. (2023) and systematized by Zhan et al. (2024).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Greshake et al. (2023) source
    • cites Zhan et al. (2024) source

      The Malicious Instruction section says Zhan et al. are the first to formalize IPI attacks on tool-integrated LLM agents, and the Introduction describes their benchmark. The paper does not say Greshake et al. introduced IPI; its Related Work discussion cites Greshake for documented real-world attacks instead. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 6.4 Additional experiments · p. 8

    In the direct hijack experiment, ASR is 80% (28 out of 40 trials) with 0% refusal rate.

    “In the direct hijack experiment, ASR is 80% (28 out of 40 trials) with 0% refusal rate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.4 Additional experiments · p. 8

    In the prompt paraphrases experiment, all 10 paraphrased queries result in successful MAS hijacking, with ASR ranging from 40% to 85%.

    “In the prompt paraphrases experiment, all 10 paraphrased queries result in successful MAS hijacking, with ASR ranging from 40% to 85%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 6.3 Varying attack payload · p. 8

    Using GPT-4o as the orchestrator improves attack success for both local and web exfiltration.

    “Using GPT-4o as the orchestrator improves attack success for both local and web exfiltration.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.4 Additional experiments · p. 8

    Direct requests to the same MAS configuration to open a reverse shell have ASR of 6% and refusal rate of 86%.

    “This is very different from the results in Table 7, where direct requests to the same MAS configuration to open a reverse shell have ASR of 6% and refusal rate of 86%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1 Introduction · p. 2

    AutoGen, CrewAI, and MetaGPT are highly vulnerable to control-flow hijacking.

    “Evaluating these systems in a variety of configurations, we demonstrate that they are highly vulnerable to control-flow hijacking.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 6.3 Varying attack payload · p. 8

    The data exfiltration attacks are effective, although slightly less so than reverse shell exploits, likely due to the number of steps involved in a successful exfiltration.

    “The attacks are effective, although slightly less so than reverse shell exploits, likely due to the number of steps involved in a successful exfiltration.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4 Control-flow hijacking in multi-agent systems · p. 5

    In MAS configurations tested, orchestrators do not directly interact with untrusted content; file-access and Web agents do, but they do not execute code directly.

    “In MAS configurations we tested, orchestrators do not directly interact with untrusted content (fileaccess and Web agents do, but they do not execute code directly).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: In all tested multi-agent topologies, each agent can see a partial history of the reasoning steps and actions that other agents have taken.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given
    § Ethics Statement · p. 11

    The MetaGPT team confirmed they were working on sandboxing their agent execution environment, and Microsoft had replicated and was investigating the issue.

    “We received confirmation that the MetaGPT team was working on sandboxing their agent execution environment, and that Microsoft had replicated and was investigating the issue.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 8 Discussion · p. 10

    Agents blindly trust the outputs of confused deputies (other agents within the multi-agent system) and find creative ways to harm the user in response to the adversary's instructions.

    “They blindly trust the outputs of confused deputies (i.e., other agents within the multi-agent system) and find creative ways to harm the user in response to the adversary's instructions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 8 Discussion · p. 10

    MAS hijacking attacks are effective across input modalities and complex adversarial tasks such as compiling and exfiltrating user profiles.

    “Sections 6.2 and 6.3 show that these attacks are effective across input modalities and complex adversarial tasks such as compiling and exfiltrating user profiles.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.5 Life finds a way · p. 8

    In one AutoGen experiment, a MAS initially tasked with describing a benign file autonomously explored its directory, discovered a malicious file, and executed it, opening a reverse shell.

    “In one AutoGen experiment, a MAS was initially tasked with describing the contents of a benign file entitled file0.txt . This benign file was in the same directory as several attack files ( file1.txt , file2.txt , etc.). After the initial task was completed, the MAS autonomously explored its directory, discovered a malicious file, and executed it, opening a reverse shell (see Appendix L).”
    Quote in the paperSubstantial claim

  • Borderline on S4 (based): unclear from the text.

    Findingnone given
    § J Refusal turns to execution (CrewAI) · p. 25

    In the CrewAI trace, refusal turns to execution.

    “J Refusal turns to execution (CrewAI)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.5 Life finds a way · p. 8

    In one AutoGen experiment, after the coder agent refused to produce malicious code, the file surfer sub-agent produced its own reverse shell, and the generated code was executed by the code executor, opening a reverse shell.

    “Similarly, in another AutoGen experiment, after the coder agent refused to produce malicious code, the file surfer sub-agent produced its own reverse shell, helpfully noting to not execute the code because it was dangerous. The generated code was then executed by the code executor, opening a reverse shell (see Appendix K).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § L Benign task turns malicious (AutoGen) · p. 29

    In the AutoGen trace, a benign task turns malicious.

    “L Benign task turns malicious (AutoGen)”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 6.5 Life finds a way · p. 9

    Although results were inconsistent, the MAS occasionally opened a reverse shell when processing a video of the attack text via audio transcription, indicating that even unconventional input modalities can serve as attack vectors.

    “Although the results were inconsistent, the MAS occasionally opened a reverse shell, indicating that even unconventional input modalities can serve as attack vectors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § E Details of experimental setup · p. 21

    GPT-4o-mini classification of output logs resulted in far too many false negatives.

    “We also attempted to use GPT 4o-mini to classify output logs, but it resulted in far too many false negatives (see Appendix M for an example).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § K Refusal turns to execution (AutoGen) · p. 27

    In the AutoGen trace, refusal turns to execution.

    “K Refusal turns to execution (AutoGen)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § A.1 AI agent security and indirect prompt injection · p. 16

    This paper systematically investigates how compromised metadata (in particular, error reports) and 'confused deputy' vulnerabilities can hijack control flows in multi-agent systems and result in malicious code execution, even when individual agents refuse to perform unsafe actions.

    “In contrast, we systematically investigate how compromised metadata (in particular, error reports) and 'confused deputy' vulnerabilities can hijack control flows in multi-agent systems and result in malicious code execution, even when individual agents refuse to perform unsafe actions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 7.1 AI agent security · p. 9

    Compromised metadata (in particular, error reports) and 'confused deputy' vulnerabilities can hijack control flows in multi-agent systems and result in malicious code execution, even when individual agents refuse to perform unsafe actions.

    “In contrast to these works, we systematically investigate how compromised metadata (in particular, error reports) and 'confused deputy' vulnerabilities can hijack control flows in multi-agent systems and result in malicious code execution, even when individual agents refuse to perform unsafe actions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingreasoning
    § 8 Discussion · p. 10

    The LLMs used to instantiate individual agents are unable to distinguish benign from malicious inputs, nor data from metadata.

    “The LLMs used to instantiate individual agents are unable to distinguish benign from malicious inputs, nor data from metadata.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § E Details of experimental setup · p. 21

    In most cases, the payload of the attack is a short reverse shell script to a remote server, written in non-obfuscated Python.

    “In most cases, the payload of the attack is a short reverse shell script to a remote server, written in non-obfuscated Python.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.5 Life finds a way · p. 8

    In several CrewAI experiments, the coding agent recognized that the reverse shell code was malicious and generated inline warnings, but failed to relay this information to the rest of the system, and the orchestrator commanded the coder to execute the code, resulting in a breach.

    “In several other CrewAI experiments, the coding agent recognized that the reverse shell code was malicious and generated inline warnings. Nevertheless, it failed to relay this information to the rest of the system. Instead, it signaled success to the orchestrator, which then commanded the coder to execute the code, ultimately resulting in a breach.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 3.2 Adversary's goals · p. 5

    Data exfiltration attack targets data that only the agents can access, whether stored in 'memory' modules, RAG databases, APIs, or derived from prior interactions.

    “This attack targets the data that only the agents can access, whether stored in 'memory' modules, RAG databases, APIs, or derived from prior interactions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 1 Introduction · p. 2

    If the multi-agent system runs in a containerized environment, any user data kept in that environment is at risk and any outputs generated from that environment are compromised.

    “If the multi-agent system runs in a containerized environment, any user data kept in that environment is at risk and any outputs generated from that environment are compromised.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § Abstract · p. 1

    MAS hijacking can result in a complete security breach, up to execution of arbitrary malicious code on the user's device or exfiltration of sensitive data from the user's containerized environment.

    “This results in a complete security breach, up to execution of arbitrary malicious code on the user's device or exfiltration of sensitive data from the user's containerized environment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.2 Adversary's goals · p. 5

    The attacker with terminal access can install malware or keyloggers, run cryptomining software, and add, remove, or copy any files the agents can access.

    “This enables the attacker to install malware or keyloggers, run cryptomining software, and add, remove, or copy any files the agents can access.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.2 Adversary's goals · p. 5

    The attacker can exploit the victim's compute resources for malicious and/or illegal activities such as hosting command-and-control beacons, forming botnets to launch distributed denial-of-service (DDoS) attacks, and running phishing campaigns.

    “The attacker can also exploit the victim's compute resources for malicious and/or illegal activities such as hosting command-and-control beacons, forming botnets to launch distributed denial-of-service (DDoS) attacks, and running phishing campaigns.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4 Control-flow hijacking in multi-agent systems · p. 5

    The MAS hijacking attack exploits inter-agent trust and leverages adaptive error-handling behavior intended to enhance the MAS's benign functionality.

    “The attack exploits inter-agent trust and leverages adaptive errorhandling behavior intended to enhance the MAS's benign functionality.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Multi-agent systems delegate tasks to individual sub-agents and trust that they correctly report the status of their tasks.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 3.2 Adversary's goals · p. 5

    By tricking the MAS into opening a reverse shell from the user's device to the adversary's device, the attacker gains terminal access to the MAS's code execution environment.

    “By tricking the MAS into opening a reverse shell from the user's device to the adversary's device, the attacker gains terminal access to the MAS's code execution environment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 3.2 Adversary's goals · p. 5

    Data exfiltration enables the attacker to extract personal data even if it is partitioned in a containerized module and isolated from the main code execution system.

    “It enables the attacker to extract personal data even if it is partitioned in a containerized module and isolated from the main code execution system.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 5 Experimental setup · p. 6

    For each experiment, outputs were logged and regex searches (along with manual analysis) were used to identify unregistered attacks, unsafe code, and refusals to act.

    “For each experiment, the outputs were logged and regex searches (along with manual analysis) were used to identify unregistered attacks (i.e., where our modified code executor did not output a unique string associated with the attack), unsafe code, and refusals to act.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 4 Control-flow hijacking in multi-agent systems · p. 5

    The MAS hijacking attack is successful because it 'launders' adversarial requests through outputs and metadata of front-line agents that directly interact with untrusted content.

    “We believe this attack is successful because it 'launders' adversarial requests through outputs and metadata of front-line agents that directly interact with untrusted content.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Multi-agent systems delegate tasks to individual sub-agents and trust that they correctly report the status of their tasks.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 8 Discussion · p. 10

    Multi-agent systems derive much of their value from their adaptive, nondeterministic control flows, which allow them to internally evaluate whether a given action successfully fulfilled the user's request and try other approaches if it has not.

    “Furthermore, multi-agent systems derive much of their value from their adaptive, nondeterministic control flows , which allow them to internally evaluate whether a given action successfully fulfilled the user's request and try other approaches if it has not.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning
    § 8 Discussion · p. 10

    Until a policy for dealing with malicious third-party content emerges, the use of LLM-based agents in situations where they deal with untrusted content and perform external actions is unsafe and has a high risk of harming their users.

    “We argue that until such a policy emerges, the use of LLM-based agents in situations where they deal with untrusted content and perform external actions is unsafe and has a high risk of harming their users.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § Ethics Statement · p. 11

    No services with live agents in production were attacked.

    “No services with live agents in production were attacked.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § E Details of experimental setup · p. 21

    A lightweight server on a cloud service continuously listened for and printed any socket connections, and all reverse shell attacks were pointed to this server's IP address.

    “As a backup, we opened a lightweight server on a cloud service that continuously listened for and printed any socket connections. All reverse shell attacks were pointed to this server's IP address.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § E Details of experimental setup · p. 21

    The code execution agent was modified to exit and write a unique, experiment-specific string to a file when it would otherwise run the attack payload.

    “To calculate the attack success rate (ASR), we modified the code execution agent to exit and write a unique, experiment-specific string to a file when it would otherwise would run the attack payload.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § E Details of experimental setup · p. 20

    The frameworks were modified as little as possible from their baseline defaults, only so that they can execute them in the environment.

    “We modified them as little as possible from their baseline defaults, only so that we can execute them in our environment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § E Details of experimental setup · p. 20

    For CrewAI, only OpenAI models were tested because Gemini models didn't work.

    “For CrewAI, only OpenAI models were tested (since Gemini models didn't work).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § Ethics Statement · p. 11

    All experiments in this paper were run in a controlled lab environment.

    “All experiments in this paper were run in a controlled lab environment.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § E Details of experimental setup · p. 21

    A combination of regex string searches and manual verification was used to derive binary values for whether each experiment contains unregistered attack execution, identification of unsafe code, and refusal to take action.

    “We saved the full output logs for all experiments, and used a combination of regex string searches and manual verification to derive binary values for whether each experiments contains an (a) unregistered attack execution (not using our unique experiment-specific string), (b) identification of unsafe code, and (c) refusal to take action.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 6.4 Additional experiments · p. 8

    The direct hijack results support the conjecture that not only the attack string must reach the orchestrator, but it must also be presented in a (previously unknown) format that evades the safety alignment of the underlying LLM.

    “This supports our conjecture that not only the attack string must reach the orchestrator, but it must also be presented in a (previously unknown) format that evades the safety alignment of the underlying LLM.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 7.1 AI agent security · p. 9

    IPI uses unsanitized results of tool calls or RAG responses to steer an LLM to take harmful action.

    “IPI uses unsanitized results of tool calls or RAG responses to steer an LLM to take harmful action.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 8 Discussion · p. 10

    LLMs do not have a security model for dealing with malicious third-party content, and it is unclear what this model or policy might look like.

    “As a result, LLMs do not have a security model for dealing with malicious third-party content, and it is unclear what this model or policy might look like.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § E Details of experimental setup · p. 20

    All experiments were run in a controlled lab environment, and no services with live agents in production were attacked.

    “All experiments were run in a controlled lab environment, and no services with live agents in production were attacked.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 8 Discussion · p. 10

    Multi-agent systems do not have a trust model, nor any semantics of inputs, actions, etc., unlike conventional operating systems and Web browsers.

    “Unlike conventional operating systems and Web browsers, they do not have a trust model, nor any semantics of inputs, actions, etc.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § E Details of experimental setup · p. 21

    The attack payload script is unambiguously unsafe.

    “This script is unambiguously unsafe.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § E Details of experimental setup · p. 21

    Full output logs were saved for all experiments.

    “We saved the full output logs for all experiments”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § 8 Discussion · p. 10

    The orchestrator's exploration of action space inevitably includes potentially unsafe actions, such as writing and executing Python programs to connect to another machine.

    “Inevitably, this involves exploration of action space, which includes potentially unsafe actions (for example, writing and executing Python programs to connect to another machine).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The orchestrator's primary job in a multi-agent system is to reason, plan, and find a way to fulfill the user's request.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisereasoning
    § 8 Discussion · p. 10

    Actions such as executing scripts are not universally unsafe; in some contexts they are necessary and appropriate.

    “In contrast to conventional safety alignment and jailbreaking, these actions are not universally unsafe. In some contexts, they are necessary and appropriate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisereasoning
    § 8 Discussion · p. 10

    For secure operation in realistic environments, the entire context in which agents make decisions matters, including the user's initial request, task ledger, other agents' behavior, etc.

    “For secure operation in realistic environments, the entire context in which agents make decisions matters, including the user's initial request, task ledger, other agents' behavior, etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 8 Discussion · p. 10

    The contexts in multi-agent systems are sufficiently complex that it is difficult to classify an action as safe or unsafe.

    “The contexts in multi-agent systems are sufficiently complex that it is difficult to classify an action as safe or unsafe.”
    Quote in the paperNot a substantial claim
54 of 54

  • The authors state this as an assumption.

    Agents in a MAS are benign and not controlled by the adversary.

    “We assume that agents in a MAS are benign and not controlled by the adversary.”

  • The authors state this as an assumption.

    The adversary cannot directly prompt the MAS, but controls the inputs that a MAS may access inadvertently or on purpose.

    “This adversary cannot directly prompt the MAS, but they control the inputs that a MAS may access inadvertently or on purpose.”

  • The paper relies on this without support.

    Untrusted content is an attack surface.

    “Untrusted content is an attack surface.”

  • The paper relies on this without support.

    The boundary between trusted and untrusted content in multi-agent systems is blurry, and it is not clear what mechanisms these systems are deploying to protect users from malicious content.

    “the boundary between trusted and untrusted content in multi-agent systems is blurry, and it is not clear what mechanisms-if any-these systems are deploying to protect users from malicious content.”

  • The paper relies on this without support.

    Multi-agent systems delegate tasks to individual sub-agents and trust that they correctly report the status of their tasks.

    “These systems delegate tasks to individual sub-agents and trust that they correctly report the status of their tasks.”

  • The paper relies on this without support.

    Metadata can influence control-flow decisions even when primary outputs are properly secured.

    “This distinction becomes particularly important when considering potential security vulnerabilities, as metadata can influence control-flow decisions even when primary outputs are properly secured.”

  • The paper relies on this without support.

    In all tested multi-agent topologies, each agent can see a partial history of the reasoning steps and actions that other agents have taken.

    “In all cases, each agent can see a partial history of the reasoning steps and actions that other agents have taken.”

  • The paper relies on this without support.

    Multi-agent systems rely on communication between imperfect agents with incomplete information.

    “Multi-agent systems rely on communication between imperfect agents with incomplete information.”

  • The paper relies on this without support.

    Multi-agent systems must be able to safely interact with both sensitive user data and external, potentially untrusted content in order to be useful as generalist AI assistants.

    “In order for multi-agent systems to be useful as generalist AI assistants, they must be able to safely interact with both sensitive user data and external, potentially untrusted content.”

  • The paper relies on this without support.

    Research on AI safety has focused heavily on protecting intelligent LLMs from adversarial users, while protecting benign users from confused LLMs has taken a back seat.

    “Research on AI safety has been focusing heavily on protecting 'intelligent' (or even 'super-intelligent') LLMs from adversarial users. Protecting benign users from confused LLMs has taken a back seat.”

  • The paper relies on this without support.

    The orchestrator's primary job in a multi-agent system is to reason, plan, and find a way to fulfill the user's request.

    “In a multi-agent system, the orchestrator's primary job is to reason, plan, and find a way to fulfill the user's request.”

  • The paper relies on this without support.

    The three frameworks AutoGen, CrewAI, and MetaGPT are prominent, open-source multi-agent systems that operate on local and remote data on users' behalf.

    “we focus on prominent, open-source, multi-agent systems can operate on both local and remote data on users' behalf.”

  • The paper relies on this without support.

    Frameworks primarily designed for simulated environments or purely conversational multi-agent interactions typically do not have the same security concerns regarding access to sensitive user resources.

    “We did not consider frameworks primarily designed for simulated environments or purely conversational multi-agent interactions, as these typically do not have the same security concerns regarding access to sensitive user resources.”
13 of 13

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

  • Direct-hijack success rate does not match the trial count

    Major

    Twenty-eight successful trials out of 40 give an attack success rate of 70%, not the reported 80%.

    Reported 80 · recomputed 70

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Norm Hardy. The Confused Deputy: (or why capabilities might have been invented). SIGOPS Oper. Syst. Rev. , 22(4):36-38, October 1988. ISSN 0163-5980. doi: 10.1145/54289. 871709. URL https://dl.acm.org/doi/10.1145/54289.871709 . · DOI

Cited by

Sorted by citations

Authors
Triedman, H., Jha, R., & Shmatikov, V.
ORCID
Triedman, Harold
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2503.12188
arXiv
2503.12188
OpenAlex
W6929645614
Semantic Scholar
6e325d0c8bf92341ef95dc8b5fa8335890c0a6c7
Cited by
0
References
43
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
Keywords
multi-agent systems, LLM agents, adversarial attacks, prompt injection, data exfiltration, containerized environments, GPT-4o

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review