Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking

Cohen, Stav, Bitton, Ron, Nassi, Ben·2024·arXiv
DOI 10.48550/arxiv.2409.08045arXiv 2409.08045
Open this paper in CoChat — free account

Abstract

In this paper, we show that with the ability to jailbreak a GenAI model, attackers can escalate the outcome of attacks against RAG-based GenAI-powered applications in severity and scale. In the first part of the paper, we show that attackers can escalate RAG membership inference attacks and RAG entity extraction attacks to RAG documents extraction attacks, forcing a more severe outcome compared to existing attacks. We evaluate the results obtained from three extraction methods, the influence of the type and the size of five embeddings algorithms employed, the size of the provided context, and the GenAI engine. We show that attackers can extract 80%-99.8% of the data stored in the database used by the RAG of a Q&A chatbot. In the second part of the paper, we show that attackers can escalate the scale of RAG data poisoning attacks from compromising a single GenAI-powered application to compromising the entire GenAI ecosystem, forcing a greater scale of damage. This is done by crafting an adversarial self-replicating prompt that triggers a chain reaction of a computer worm within the ecosystem and forces each affected application to perform a malicious activity and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of confidential data extraction about users within a GenAI ecosystem of GenAI-powered email assistants and analyze how the performance of the worm is affected by the size of the context, the adversarial self-replicating prompt used, the type and size of the embeddings algorithm employed, and the number of hops in the propagation. Finally, we review and analyze guardrails to protect RAG-based inference and discuss the tradeoffs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2409.08045

Cites

Statistics

Figures

Figure 1 The influence of various extraction methods (top) and context size k (bottom) on the extraction rates.
Figure 1 The influence of various extraction methods (top) and context size k (bottom) on the extraction rates.CC-BY-NC-ND · From the PDF
Figure 2 The influence of various embeddings algorithms and sizes (top) and GenAI engine (bottom).
Figure 2 The influence of various embeddings algorithms and sizes (top) and GenAI engine (bottom).CC-BY-NC-ND · From the PDF
Figure 3 A RAG-based GenAI worm propagates from u 1 to u 2 to u 3.
Figure 3 A RAG-based GenAI worm propagates from u 1 to u 2 to u 3.CC-BY-NC-ND · From the PDF
Figure 4 The influence of the prefix of the worm (top) and the embeddings algorithm used (bottom).
Figure 4 The influence of the prefix of the worm (top) and the embeddings algorithm used (bottom).CC-BY-NC-ND · From the PDF
Figure 5 The retrieval success rate, replication success rate, replication & payload success rate and combined success rate for the three propagation ways of the worm: via a generated a new email based on subject, via the enrichment of content of a given email body, and via a generated response.
Figure 5 The retrieval success rate, replication success rate, replication & payload success rate and combined success rate for the three propagation ways of the worm: via a generated a new email based on subject, via the enrichment of content of a given email body, and via a generated response.CC-BY-NC-ND · From the PDF
Figure 7 The influence of the number of hops of the propagation (top) and the GenAI engine employed (bottom).
Figure 7 The influence of the number of hops of the propagation (top) and the GenAI engine employed (bottom).CC-BY-NC-ND · From the PDF
Figure 9 The Payload Performance (top) and Email Addresses Count in (bottom).
Figure 9 The Payload Performance (top) and Email Addresses Count in (bottom).CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

5 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 42

  1. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rock- täschel, et al., "Retrieval-augmented generation for knowledge-intensive nlp tasks," Advances in Neural In- formation Processing Systems, vol. 33, pp. 9459-9474, 2020.
  2. M. Anderson, G. Amit, and A. Goldsteen, "Is my data in your retrieval database? membership inference attacks against retrieval augmented generation," arXiv preprint arXiv:2405.20446, 2024. · arXiv
  3. Y. Li, G. Liu, Y. Yang, and C. Wang, "Seeing is believing: Black-box membership inference attacks against retrieval augmented generation," arXiv preprint arXiv:2406.19234, 2024. · arXiv
  4. S. Zeng, J. Zhang, P. He, Y. Xing, Y. Liu, H. Xu, J. Ren, S. Wang, D. Yin, Y. Chang, et al., "The good and the bad: Exploring privacy issues in retrieval-augmented genera- tion (rag)," arXiv preprint arXiv:2402.16893, 2024. · arXiv
  5. J. Xue, M. Zheng, Y. Hu, F. Liu, X. Chen, and Q. Lou, "Badrag: Identifying vulnerabilities in retrieval aug- mented generation of large language models," arXiv preprint arXiv:2406.00083, 2024. · arXiv
  6. P. Cheng, Y. Ding, T. Ju, Z. Wu, W. Du, P. Yi, Z. Zhang, and G. Liu, "Trojanrag: Retrieval-augmented generation can be backdoor driver in large language models," arXiv preprint arXiv:2405.13401, 2024. · arXiv
  7. W. Zou, R. Geng, B. Wang, and J. Jia, "Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language models," arXiv preprint arXiv:2402.07867, 2024. · arXiv
  8. A. Shafran, R. Schuster, and V. Shmatikov, "Ma- chine against the rag: Jamming retrieval-augmented generation with blocker documents," arXiv preprint arXiv:2406.05870, 2024. · arXiv
  9. H. Chaudhari, G. Severi, J. Abascal, M. Jagielski, C. A. Choquette-Choo, M. Nasr, C. Nita-Rotaru, and A. Oprea, "Phantom: General trigger attacks on re- trieval augmented language generation," arXiv preprint arXiv:2405.20485, 2024. · arXiv
  10. X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, "" do anything now": Characterizing and evaluating in- the-wild jailbreak prompts on large language models," arXiv preprint arXiv:2308.03825, 2023. · arXiv
  11. Z. Yu, X. Liu, S. Liang, Z. Cameron, C. Xiao, and N. Zhang, "Don't listen to me: Understanding and ex- ploring jailbreak prompts of large language models," arXiv preprint arXiv:2403.17336, 2024. · arXiv
  12. A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, "Universal and transferable adversarial attacks on aligned language models," arXiv preprint arXiv:2307.15043, 2023. · arXiv
  13. Y. Yang, B. Hui, H. Yuan, N. Gong, and Y. Cao, "Sneakyprompt: Jailbreaking text-to-image generative models," arXiv preprint arXiv:2305.12082, 2023. · arXiv
  14. Z.-X. Yong, C. Menghini, and S. H. Bach, "Low- resource languages jailbreak gpt-4," arXiv preprint arXiv:2310.02446, 2023. · arXiv
  15. X. Liu, N. Xu, M. Chen, and C. Xiao, "Autodan: Generat- ing stealthy jailbreak prompts on aligned large language models," arXiv preprint arXiv:2310.04451, 2023. · arXiv
  16. A. Wei, N. Haghtalab, and J. Steinhardt, "Jailbroken: How does llm safety training fail?," Advances in Neural Information Processing Systems, vol. 36, 2024.
  17. J. Hayase, E. Borevkovic, N. Carlini, F. Tramèr, and M. Nasr, "Query-based adversarial prompt generation," arXiv preprint arXiv:2402.12329, 2024. · arXiv
  18. Y. Li, Z. Li, K. Zhang, R. Dan, S. Jiang, and Y. Zhang, "Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge," Cureus, vol. 15, no. 6, 2023.
  19. B. Klimt and Y. Yang, "The enron corpus: A new dataset for email classification research," in European confer- ence on machine learning, pp. 217-226, Springer, 2004.
  20. K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu, "Mpnet: Masked and permuted pre-training for language under- standing," arXiv preprint arXiv:2004.09297, 2020. · arXiv
  21. Y. Zhang and D. Ippolito, "Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success," arXiv preprint arXiv:2307.06865, 2023. · arXiv
  22. B. Hui, H. Yuan, N. Gong, P. Burlina, and Y. Cao, "Pleak: Prompt leaking attacks against large language model applications," arXiv preprint arXiv:2405.06823, 2024. · arXiv
  23. Z. Sha and Y. Zhang, "Prompt stealing attacks against large language models," arXiv preprint arXiv:2402.12959, 2024. · arXiv
  24. J. X. Morris, W. Zhao, J. T. Chiu, V. Shmatikov, and A. M. Rush, "Language model inversion," arXiv preprint arXiv:2311.13647, 2023. · arXiv
  25. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al., "Extracting training data from large language models," in 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
  26. R. Shokri, M. Stronati, C. Song, and V. Shmatikov, "Membership inference attacks against machine learn- ing models," in 2017 IEEE symposium on security and privacy (SP), pp. 3-18, IEEE, 2017.
  27. M. Fredrikson, S. Jha, and T. Ristenpart, "Model inver- sion attacks that exploit confidence information and ba- sic countermeasures," in Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1322-1333, 2015.
  28. D. M. Kienzle and M. C. Elder, "Recent worms: a survey and trends," in Proceedings of the 2003 ACM workshop on Rapid Malcode, pp. 1-10, 2003.
  29. N. Weaver, V. Paxson, S. Staniford, and R. Cunningham, "A taxonomy of computer worms," in Proceedings of the 2003 ACM workshop on Rapid Malcode, pp. 11-18, 2003.
  30. C. Smith, A. Matrawy, S. Chow, and B. Abdelaziz, "Computer worms: Architectures, evasion strategies, and detection mechanisms," Journal of Information Assur- ance and Security, vol. 4, pp. 69-83, 2009.
  31. J. F. Shoch and J. A. Hupp, "The "worm" pro- grams-early experience with a distributed computa- tion," Communications of the ACM, vol. 25, no. 3, pp. 172-180, 1982.
  32. C. Kelty, "The morris worm," Limn, vol. 1, no. 1, 2011.
  33. H. Orman, "The morris worm: A fifteen-year perspec- tive," IEEE Security & Privacy, vol. 1, no. 5, pp. 35-43, 2003.
  34. N. Falliere, L. O. Murchu, and E. Chien, "W32. stuxnet dossier," White paper, symantec corp., security response, vol. 5, no. 6, p. 29, 2011.
  35. D. Kushner, "The real story of stuxnet," ieee Spectrum, vol. 50, no. 3, pp. 48-53, 2013.
  36. A. Matrosov, E. Rodionov, D. Harley, and J. Malcho, "Stuxnet under the microscope," ESET LLC (September 2010), vol. 6, 2010.
  37. M. Antonakakis, T. April, M. Bailey, M. Bernhard, E. Bursztein, J. Cochran, Z. Durumeric, J. A. Halder- man, L. Invernizzi, M. Kallitsis, et al., "Understanding the mirai botnet," in 26th USENIX security symposium (USENIX Security 17), pp. 1093-1110, 2017.
  38. M. Akbanov, V. G. Vassilakis, and M. D. Logothetis, "Wannacry ransomware: Analysis of infection, persis- tence, recovery prevention and propagation mecha- nisms," Journal of Telecommunications and Information Technology, no. 1, pp. 113-124, 2019.
  39. Q. Chen and R. A. Bridges, "Automated behavioral anal- ysis of malware: A case study of wannacry ransomware," in 2017 16th IEEE International Conference on machine learning and applications (ICMLA), pp. 454-460, IEEE, 2017.
  40. D.-Y. Kao and S.-C. Hsiao, "The dynamic analysis of wannacry ransomware," in 2018 20th International conference on advanced communication technology (ICACT), pp. 159-166, IEEE, 2018.
  41. S.-C. Hsiao and D.-Y. Kao, "The static analysis of wan- nacry ransomware," in 2018 20th international confer- ence on advanced communication technology (ICACT), pp. 153-158, IEEE, 2018.
  42. Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang, "Towards general text embeddings with multi-stage contrastive learning," arXiv preprint arXiv:2308.03281, 2023. · arXiv

Cited by

See all 1

Metadata

Authors
Cohen, S., Bitton, R., & Nassi, B.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2409.08045
arXiv
2409.08045
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking. https://app.cochat.ai/papers/10.48550/arxiv.2409.08045
The paper
Cohen, S., Bitton, R., & Nassi, B. (2024). Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking. arXiv. https://doi.org/10.48550/arxiv.2409.08045