Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systems
Open accessAbstract
As AI agents are increasingly adopted to collaborate on complex objectives, ensuring the security of autonomous multi-agent systems becomes crucial. We develop simulations of agents collaborating on shared objectives to study these security risks and security trade-offs. We focus on scenarios where an attacker compromises one agent, using it to steer the entire system toward misaligned outcomes by corrupting other agents. In this context, we observe infectious malicious prompts - the multi-hop spreading of malicious instructions. To mitigate this risk, we evaluated several strategies: two "vaccination" approaches that insert false memories of safely handling malicious input into the agents' memory stream, and two versions of a generic safety instruction strategy. While these defenses reduce the spread and fulfillment of malicious instructions in our experiments, they tend to decrease collaboration capability in the agent network. Our findings illustrate potential trade-off between security and collaborative efficiency in multi-agent systems, providing insights for designing more secure yet effective AI collaborations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Network effects can rapidly spread privacy breaches, disinformation, jailbreaks, and data poisoning.
“Network effects can rapidly spread privacy breaches, disinformation, jailbreaks, and data poisoning”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Peigné et al., 2025
The Introduction says malicious prompts can spread rapidly between agents, and the Figure 2 description shows a jailbreak propagating through messages. The Discussion & Limitations says private-data exfiltration was not studied; our copy does not establish the claimed spread of privacy breaches, disinformation, or data poisoning. Page numbers are not available in the supplied passages. (LLM, conf=0.88)
Experimental — accuracy not yet measuredAs agent systems scale, network effects can amplify vulnerabilities (cascading privacy leaks, jailbreaks or prompt injections proliferating across agent boundaries) or enable coordination of adversarial behaviors that evade detection.
“As agent systems scale, network effects can amplify vulnerabilities -cascading privacy leaks, jailbreaks or prompt injections proliferating across agent boundaries (Peigné et al., 2025;Naik et al., 2026), or enabling coordination of adversarial behaviors against agents, platforms, humans and institutions that evade detection.”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Peigné et al., 2025
The Introduction and Discussion & Limitations describe malicious prompts spreading across agents through communication channels, including multi-hop proliferation of jailbreak instructions. But Discussion & Limitations says private-data exfiltration was a scenario not studied, so the cited paper does not establish cascading privacy leaks. The claim about adversarial coordination appears after the citation in the printed sentence and is not assessed as attributed to this source. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe authors observed that system robustness (no explosion) rose from 76.7% under no defense to 83.3% with passive vaccines and 90.0% with active vaccines, while agent cooperation remained high (87.5% passive, 87.6% active) versus 87.1% without defense.
“The authors observed that system robustness (no explosion) rose from 76.7 % under no defense to 83.3 % with passive vaccines and 90.0 % with active vaccines, while agent cooperation, i.e. acceptance of benign but unusual requests, remained high (87.5 % passive, 87.6 % active) versus 87.1 % without defense.”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Peigné et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Emergent autonomous scientific research capabilities of large language models
Boiko, D. A., MacKnight, R., & Gomes, G. (2023) · arXiv · doi:10.48550/arxiv.2304.05332 · Cited by 72 · see citing works
Flagged first, then by citations
Research works printed but not linked: 33
- Andriushchenko, M.; Souly, A.; Dziemian, M.; Duenas, D.; Lin, M.; Wang, J.; Hendrycks, D.; Zou, A.; Kolter, Z.; Fredrikson, M.; Winsor, E.; Wynne, J.; Gal, Y.; and Davies, X. 2024. AgentHarm: A Benchmark for Measuring Harm- fulness of LLM Agents. ArXiv.
- Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165. · arXiv
- Cohen, S.; Bitton, R.; and Nassi, B. 2024. Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI- Powered Applications. arXiv:2403.02817. · arXiv
- Greshake, K.; Abdelnabi, S.; Mishra, S.; Endres, C.; Holz, T.; and Fritz, M. 2023. Not what you've signed up for: Com- promising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173. · arXiv
- Gu, X.; Zheng, X.; Pang, T.; Du, C.; Liu, Q.; Wang, Y.; Jiang, J.; and Lin, M. 2024. Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Expo- nentially Fast. arXiv:2402.08567. · arXiv
- Hao, S.; Gu, Y.; Ma, H.; Hong, J. J.; Wang, Z.; Wang, D. Z.; and Hu, Z. 2023. Reasoning with Language Model is Plan- ning with World Model. arXiv:2305.14992. · arXiv
- Hong, S.; Zhuge, M.; Chen, J.; Zheng, X.; Cheng, Y.; Zhang, C.; Wang, J.; Wang, Z.; Yau, S. K. S.; Lin, Z.; Zhou, L.; Ran, C.; Xiao, L.; Wu, C.; and Schmidhuber, J. 2023. MetaGPT: Meta Programming for A Multi-Agent Collaborative Frame- work. arXiv:2308.00352. · arXiv
- Hua, W.; Yang, X.; Jin, M.; Li, Z.; Cheng, W.; Tang, R.; and Zhang, Y. 2024. TrustAgent: Towards Safe and Trustworthy LLM-based Agents. arXiv:2402.01586. · arXiv
- Hughes, J.; Price, S.; Lynch, A.; Schaeffer, R.; Barez, F.; Koyejo, S.; Sleight, H.; Jones, E.; Perez, E.; and Sharma, M. 2024. Best-of-N Jailbreaking. arXiv:2412.03556. · arXiv
- Kim, J.; Derakhshan, A.; and Harris, I. G. 2023. Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield. arXiv:2311.00172. · arXiv
- Kumar, P.; Lau, E.; Vijayakumar, S.; Trinh, T.; Team, S. R.; Chang, E.; Robinson, V.; Hendryx, S.; Zhou, S.; Fredrikson, M.; Yue, S.; and Wang, Z. 2024. Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents. ArXiv.
- Langner, R. 2011. Stuxnet: Dissecting a Cyberwarfare Weapon. IEEE Security & Privacy, 9(3): 49-51.
- Lermen, S.; Dziemian, M.; and Pimpale, G. 2024. Applying Refusal-Vector Ablation to Llama 3.1 70B Agents. ArXiv.
- Li, H.; Chen, Y.; Luo, J.; Kang, Y.; Zhang, X.; Hu, Q.; Chan, C.; and Song, Y. 2023. Privacy in Large Lan- guage Models: Attacks, Defenses and Future Directions. arXiv:2310.10383. · arXiv
- Li, H.; Su, Y.; Cai, D.; Wang, Y.; and Liu, L. 2022. A Survey on Retrieval-Augmented Text Generation. arXiv:2202.01110. · arXiv
- Liang, Y.; Wu, C.; Song, T.; Wu, W.; Xia, Y.; Liu, Y.; Ou, Y.; Lu, S.; Ji, L.; Mao, S.; Wang, Y.; Shou, L.; Gong, M.; and Duan, N. 2023. TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs. arXiv:2303.16434. · arXiv
- Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H.; and Neubig, G. 2021. Pre-train, Prompt, and Predict: A Systematic Sur- vey of Prompting Methods in Natural Language Processing. arXiv:2107.13586. · arXiv
- Liu, Y.; Deng, G.; Li, Y.; Wang, K.; Wang, Z.; Wang, X.; Zhang, T.; Liu, Y.; Wang, H.; Zheng, Y.; and Liu, Y. 2024. Prompt Injection attack against LLM-integrated Applica- tions. arXiv:2306.05499. Llama team and contributors. 2023. LLaMA: Open and Ef- ficient Foundation Language Models. arXiv:2302.13971. · arXiv
- Nascimento, N.; Alencar, P.; and Cowan, D. 2023. Self- Adaptive Large Language Model (LLM)-Based Multiagent Systems. arXiv:2307.06187. OpenAI. 2023. ChatGPT plugins. https://openai.com/index/ chatgpt-plugins/. Accessed: 2024-06-15. OpenAI. 2024a. GPT-4 Technical Report. arXiv:2303.08774. · arXiv
- OpenAI. 2024b. GPT-4o System Card. arXiv:2410.21276. · arXiv
- Park, J. S.; O'Brien, J. C.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative Agents: Interac- tive Simulacra of Human Behavior. ArXiv:2304.03442 · arXiv
- Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; Zhao, S.; Hong, L.; Tian, R.; Xie, R.; Zhou, J.; Gerstein, M.; Li, D.; Liu, Z.; and Sun, M. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. ArXiv:2307.16789 · arXiv
- Shen, Y.; Song, K.; Tan, X.; Li, D.; Lu, W.; and Zhuang, Y. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. arXiv:2303.17580. · arXiv
- Tian, Y.; Yang, X.; Zhang, J.; Dong, Y.; and Su, H. 2024. Evil Geniuses: Delving into the Safety of LLM-based Agents. arXiv:2311.11855. · arXiv
- Wang, L.; Ma, C.; Feng, X.; Zhang, Z.; Yang, H.; Zhang, J.; Chen, Z.; Tang, J.; Chen, X.; Lin, Y.; Zhao, W. X.; Wei, Z.; and Wen, J.-R. 2024a. A Survey on Large Language Model based Autonomous Agents. Frontiers of Computer Science, 18(6): 186345. ArXiv:2308.11432 · arXiv
- Wang, Z.; Mao, S.; Wu, W.; Ge, T.; Wei, F.; and Ji, H. 2024b. Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration. arXiv:2307.05300. · arXiv
- Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems, volume 36, 80079-80110.
- Yang, W.; Bi, X.; Lin, Y.; Chen, S.; Zhou, J.; and Sun, X. 2024. Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents. arXiv:2402.11208. · arXiv
- Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T. L.; Cao, Y.; and Narasimhan, K. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601. · arXiv
- Zhang, H.; Du, W.; Shan, J.; Zhou, Q.; Du, Y.; Tenenbaum, J. B.; Shu, T.; and Gan, C. 2024. Building Cooperative Embodied Agents Modularly with Large Language Models. arXiv:2307.02485. · arXiv
- Zhao, A.; Huang, D.; Xu, Q.; Lin, M.; Liu, Y.-J.; and Huang, G. 2023. ExpeL: LLM Agents Are Experiential Learners. arXiv:2308.10144. · arXiv
- Zheng, Z.; Zhang, O.; Nguyen, H. L.; Rampal, N.; Alawadhi, A. H.; Rong, Z.; Head-Gordon, T.; Borgs, C.; Chayes, J. T.; and Yaghi, O. M. 2023. ChatGPT Research Group for Optimizing the Crystallinity of MOFs and COFs. ACS Central Science, 9(11): 2161-2170.
- Zou, A.; Phan, L.; Wang, J.; Duenas, D.; Lin, M.; An- driushchenko, M.; Wang, R.; Kolter, Z.; Fredrikson, M.; and Hendrycks, D. 2024. Improving Alignment and Robustness with Circuit Breakers. arXiv:2406.04313. · arXiv
Cited by
Sorted by citations
Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 2 · see citing works
SoK: The Attack Surface of Agentic AI - Tools and Autonomy
Dehghantanha, A., & Homayoun, S. (2026) · arXiv · doi:10.48550/arxiv.2603.22928
- Authors
- Peigné, P., Kniejski, M., Sondej, F., David, M., Hoelscher-Obermaier, J., Schroeder de Witt, C., & Kran, E.
- Affiliations
- Protéomique, Réponse Inflammatoire et Spectrométrie de Masse (FR)
- Jagiellonian University (PL)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2025-04-11
- Volume
- 39
- Issue
- 26
- Pages
- 27573-27581
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4409361695
- Cited by
- 8
- References
- 1
- Citations by year
- 2026: 6 · 2025: 2
- FWCI
- 7.59
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- Technische Universiteit Delft
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Sociology and Political Science › Access Control and Trust
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- multi-agent systems, AI agent security, vaccination strategies, safety instructions, misalignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
