CoChat
Sign Up
PreprintCited by 10Adversarial Robustness in Machine LearningSecurity and Verification in ComputingWeb Application Security Vulnerabilities

Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications

Open access
Cohen, Stav, Bitton, Ron, Nassi, Ben·2024·arXiv
DOI 10.48550/arxiv.2403.02817arXiv 2403.02817
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we show that when the communication between GenAI-powered applications relies on RAG-based inference, an attacker can initiate a computer worm-like chain reaction that we call Morris-II. This is done by crafting an adversarial self-replicating prompt that triggers a cascade of indirect prompt injections within the ecosystem and forces each affected application to perform malicious actions and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of confidential user data extraction within a GenAI ecosystem of GenAI-powered email assistants and analyze how the performance of the worm is affected by the size of the context, the adversarial self-replicating prompt used, the type and size of the embedding algorithm employed, and the number of hops in the propagation. Finally, we introduce the Virtual Donkey, a guardrail intended to detect and prevent the propagation of Morris-II with minimal latency, high accuracy, and a low false-positive rate. We evaluate the guardrail's performance and show that it yields a perfect true-positive rate of 1.0 with a false-positive rate of 0.015, and is robust against out-of-distribution worms, consisting of unseen jailbreaking commands, a different email dataset, and various worm usecases.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.02817

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent work has begun to reveal that similar cascade-based threats can compromise networks of LLM agents, spreading malicious behavior across cooperative populations with alarming speed and stealth.

    “Recent work has begun to reveal that similar cascade-based threats can compromise networks of LLM agents, spreading malicious behavior across cooperative populations with alarming speed and stealth (Cohen et al., 2024;Ju et al., 2024;Gu et al., 2024;Lee & Tiwari, 2024a;Peigné et al., 2025).”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Cohen et al., 2024

    The Introduction (p. 1) describes a worm-like chain reaction that makes affected GenAI applications perform malicious actions and compromise additional applications. But the studied setting is an ecosystem of RAG-based applications, including email assistants—not cooperative populations of LLM agents—and the cited passages do not establish alarming speed or stealth. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 37

  1. [1][1] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. , 'Retrieval-augmented generation for knowledge-intensive nlp tasks,' Advances in Neural Information Processing Systems , vol. 33, pp. 94599474, 2020. -
  2. [2][2] M. Anderson, G. Amit, and A. Goldsteen, 'Is my data in your retrieval database? membership inference attacks against retrieval augmented generation,' arXiv preprint arXiv:2405.20446 , 2024. - · arXiv
  3. [3][3] Y. Li, G. Liu, Y. Yang, and C. Wang, 'Seeing is believing: Black-box membership inference attacks against retrieval augmented generation,' arXiv preprint arXiv:2406.19234 , 2024. - · arXiv
  4. [4][4] S. Zeng, J. Zhang, P. He, Y. Xing, Y. Liu, H. Xu, J. Ren, S. Wang, D. Yin, Y. Chang, et al. , 'The good and the bad: Exploring privacy issues in retrieval-augmented generation (rag),' arXiv preprint arXiv:2402.16893 , 2024. - · arXiv
  5. [5][5] J. Xue, M. Zheng, Y. Hu, F. Liu, X. Chen, and Q. Lou, 'Badrag: Identifying vulnerabilities in retrieval augmented generation of large language models,' arXiv preprint arXiv:2406.00083 , 2024. - · arXiv
  6. [6][6] P. Cheng, Y. Ding, T. Ju, Z. Wu, W. Du, P. Yi, Z. Zhang, and G. Liu, 'Trojanrag: Retrieval-augmented generation can be backdoor driver in large language models,' arXiv preprint arXiv:2405.13401 , 2024. - · arXiv
  7. [7][7] W. Zou, R. Geng, B. Wang, and J. Jia, 'Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language models,' arXiv preprint arXiv:2402.07867 , 2024. - · arXiv
  8. [8][8] A. Shafran, R. Schuster, and V. Shmatikov, 'Machine against the rag: Jamming retrieval-augmented generation with blocker documents,' arXiv preprint arXiv:2406.05870 , 2024. - · arXiv
  9. [9][9] H. Chaudhari, G. Severi, J. Abascal, M. Jagielski, C. A. Choquette-Choo, M. Nasr, C. Nita-Rotaru, and A. Oprea, 'Phantom: General trigger attacks on retrieval augmented language generation,' arXiv preprint arXiv:2405.20485 , 2024. - · arXiv
  10. [10][10] C. Kelty, 'The morris worm,' Limn , vol. 1, no. 1, 2011. -
  11. [12][12] H. Orman, 'The morris worm: A fifteen-year perspective,' IEEE Security & Privacy , vol. 1, no. 5, pp. 3543, 2003. -
  12. [13][13] B. Klimt and Y. Yang, 'The enron corpus: A new dataset for email classification research,' in European conference on machine learning , pp. 217-226, Springer, 2004. -
  13. [14][14] K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu, 'Mpnet: Masked and permuted pre-training for language understanding,' arXiv preprint arXiv:2004.09297 , 2020. - · arXiv
  14. [15][15] D. M. Kienzle and M. C. Elder, 'Recent worms: a survey and trends,' in Proceedings of the 2003 ACM workshop on Rapid Malcode , pp. 1-10, 2003. -
  15. [16][16] N. Weaver, V. Paxson, S. Staniford, and R. Cunningham, 'A taxonomy of computer worms,' in Proceedings of the 2003 ACM workshop on Rapid Malcode , pp. 11-18, 2003. -
  16. [17][17] C. Smith, A. Matrawy, S. Chow, and B. Abdelaziz, 'Computer worms: Architectures, evasion strategies, and detection mechanisms,' Journal of Information Assurance and Security , vol. 4, pp. 69-83, 2009. -
  17. [18][18] J. F. Shoch and J. A. Hupp, 'The 'worm' programs-early experience with a distributed computation,' Communications of the ACM , vol. 25, no. 3, pp. 172-180, 1982. -
  18. [21][21] N. Falliere, L. O. Murchu, and E. Chien, 'W32. stuxnet dossier,' White paper, symantec corp., security response , vol. 5, no. 6, p. 29, 2011. -
  19. [22][22] D. Kushner, 'The real story of stuxnet,' ieee Spectrum , vol. 50, no. 3, pp. 48-53, 2013. -
  20. [23][23] A. Matrosov, E. Rodionov, D. Harley, and J. Malcho, 'Stuxnet under the microscope,' ESET LLC (September 2010) , vol. 6, 2010. -
  21. [24][24] M. Antonakakis, T. April, M. Bailey, M. Bernhard, E. Bursztein, J. Cochran, Z. Durumeric, J. A. Halderman, L. Invernizzi, M. Kallitsis, et al. , 'Understanding the mirai botnet,' in 26th USENIX security symposium (USENIX Security 17) , pp. 1093-1110, 2017. -
  22. [25][25] M. Akbanov, V. G. Vassilakis, and M. D. Logothetis, 'Wannacry ransomware: Analysis of infection, persistence, recovery prevention and propagation mechanisms,' Journal of Telecommunications and Information Technology , no. 1, pp. 113-124, 2019. -
  23. [26][26] Q. Chen and R. A. Bridges, 'Automated behavioral analysis of malware: A case study of wannacry ransomware,' in 2017 16th IEEE International Conference on machine learning and applications (ICMLA) , pp. 454-460, IEEE, 2017. -
  24. [27][27] D.-Y. Kao and S.-C. Hsiao, 'The dynamic analysis of wannacry ransomware,' in 2018 20th International conference on advanced communication technology (ICACT) , pp. 159-166, IEEE, 2018. -
  25. [28][28] S.-C. Hsiao and D.-Y. Kao, 'The static analysis of wannacry ransomware,' in 2018 20th international conference on advanced communication technology (ICACT) , pp. 153-158, IEEE, 2018. -
  26. [29][29] N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, A. Awadalla, P. W. Koh, D. Ippolito, K. Lee, F. Tramer, et al. , 'Are aligned neural networks adversarially aligned?,' arXiv preprint arXiv:2306.15447 , 2023. - · arXiv
  27. [30][30] E. Bagdasaryan, T.-Y. Hsieh, B. Nassi, and V. Shmatikov, '(ab) using images and sounds 31. for indirect instruction injection in multi-modal llms,' arXiv preprint arXiv:2307.10490 , 2023. - · arXiv
  28. [31][31] F. Perez and I. Ribeiro, 'Ignore previous prompt: Attack techniques for language models,' arXiv preprint arXiv:2211.09527 , 2022. - · arXiv
  29. [32][32] S. Abdelnabi, K. Greshake, S. Mishra, C. Endres, T. Holz, and M. Fritz, 'Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,' in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security , pp. 79-90, 2023. -
  30. [33][33] F. Jiang, Z. Xu, L. Niu, Z. Xiang, B. Ramasubramanian, B. Li, and R. Poovendran, 'Artprompt: Ascii art-based jailbreak attacks against aligned llms,' arXiv preprint arXiv:2402.11753 , 2024. - · arXiv
  31. [35][35] X. Liu, Z. Yu, Y. Zhang, N. Zhang, and C. Xiao, 'Automatic and universal prompt injection attacks against large language models,' arXiv preprint arXiv:2403.04957 , 2024. - · arXiv
  32. [36][36] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, '" do anything now": Characterizing and evaluating inthe-wild jailbreak prompts on large language models,' arXiv preprint arXiv:2308.03825 , 2023. - · arXiv
  33. [37][37] B. Hui, H. Yuan, N. Gong, P. Burlina, and Y. Cao, 'Pleak: Prompt leaking attacks against large language model applications,' arXiv preprint arXiv:2405.06823 , 2024. - · arXiv
  34. [38][38] Z. Sha and Y. Zhang, 'Prompt stealing attacks against large language models,' arXiv preprint arXiv:2402.12959 , 2024. - · arXiv
  35. [39][39] R. Weiss, D. Ayzenshteyn, and Y. Mirsky, 'What was your prompt? a remote keylogging attack on AI assistants,' in 33rd USENIX Security Symposium (USENIX Security 24) , (Philadelphia, PA), pp. 33673384, USENIX Association, Aug. 2024. -
  36. [40][40] K. Gao, Y. Bai, J. Gu, S.-T. Xia, P. Torr, Z. Li, and W. Liu, 'Inducing high energy-latency of large visionlanguage models with verbose images,' arXiv preprint arXiv:2401.11170 , 2024. - · arXiv
  37. [45][45] Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang, 'Towards general text embeddings with multi-stage contrastive learning,' arXiv preprint arXiv:2308.03281 , 2023. · arXiv

Cited by

Sorted by citations

Related

Authors
Cohen, S., Bitton, R., & Nassi, B.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.02817
arXiv
2403.02817
OpenAlex
W4392538361
Cited by
4
References
0
Citations by year
2026: 1 · 2025: 2 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
Keywords
generative AI security, prompt injection, adversarial prompts, worm propagation, jailbreaking, out-of-distribution robustness, embedding algorithms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review