PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
Open accessAbstract
Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. The key idea of RAG is to ground the answer generation of an LLM on external knowledge retrieved from a knowledge database. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG, where an attacker could inject a few malicious texts into the knowledge database of a RAG system to induce an LLM to generate an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge corruption attacks as an optimization problem, whose solution is a set of malicious texts. Depending on the background knowledge (e.g., black-box and white-box settings) of an attacker on a RAG system, we propose two solutions to solve the optimization problem, respectively. Our results show PoisonedRAG could achieve a 90% attack success rate when injecting five malicious texts for each target question into a knowledge database with millions of texts. We also evaluate several defenses and our results show they are insufficient to defend against PoisonedRAG, highlighting the need for new defenses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 44
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv, 2023.
- R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen et al., "Palm 2 technical report," arXiv, 2023.
- R. Z. Mahari, "Autolaw: Augmented legal reasoning through legal precedent prediction," arXiv, 2021.
- J. Boyko, J. Cohen, N. Fox, M. H. Veiga, J. I. Li, J. Liu, B. Modenesi, A. H. Rauch, K. N. Reid, S. Tribedi et al., "An interdisciplinary outlook on large language models for scientific research," arXiv, 2023.
- P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rock- täschel et al., "Retrieval-augmented generation for knowledge-intensive nlp tasks," NeurIPS, 2020.
- S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark et al., "Improving language models by retrieving from trillions of tokens," in ICML, 2022.
- R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kul- shreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du et al., "Lamda: Language models for dialog applica- tions," arXiv, 2022.
- N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych, "Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models," in NeurIPS, 2021.
- S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, "React: Synergiz- ing reasoning and acting in language models," arXiv, 2022.
- A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, "Self-rag: Learning to retrieve, generate, and critique through self-reflection," in ICLR, 2024.
- G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bo- janowski, A. Joulin, and E. Grave, "Unsupervised dense information retrieval with contrastive learning," Transactions on Machine Learning Research, 2022.
- L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. N. Ben- nett, J. Ahmed, and A. Overwijk, "Approximate nearest neighbor negative contrastive learning for dense text retrieval," in ICLR, 2021.
- Z. Peng, X. Wu, and Y. Fang, "Soft prompt tuning for augmenting dense retrieval with large language models," arXiv, 2023.
- V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, "Dense passage retrieval for open-domain question answering," in EMNLP, 2020.
- T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng, "Ms marco: A human generated machine reading comprehension dataset," choice, vol. 2640, p. 660, 2016.
- H. Touvron, L. Martin, K. Stone, P. Albert, A. Alma- hairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine- tuned chat models," arXiv, 2023.
- N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, "Baseline defenses for adversarial attacks against aligned language models," arXiv, 2023.
- G. Alon and M. Kamfonas, "Detecting language model attacks with perplexity," arXiv, 2023.
- F. Perez and I. Ribeiro, "Ignore previous prompt: At- tack techniques for language models," in NeurIPS ML Safety Workshop, 2022.
- Y. Liu, G. Deng, Y. Li, K. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, and Y. Liu, "Prompt injection at- tack against llm-integrated applications," arXiv, 2023.
- R. Pedro, D. Castro, P. Carreira, and N. Santos, "From prompt injections to sql injection attacks: How pro- tected is your llm-integrated web application?" arXiv, 2023.
- H. J. Branch, J. R. Cefalu, J. McHugh, L. Hujer, A. Bahl, D. d. C. Iglesias, R. Heichman, and R. Dar- wishi, "Evaluating the susceptibility of pre-trained lan- guage models via handcrafted adversarial examples," arXiv, 2022.
- A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, "Uni- versal and transferable adversarial attacks on aligned language models," arXiv, 2023.
- N. Kandpal, M. Jagielski, F. Tramèr, and N. Carlini, "Backdoor attacks for in-context learning with language models," in ICML Workshop, 2023.
- A. Wan, E. Wallace, S. Shen, and D. Klein, "Poisoning language models during instruction tuning," in ICML, 2023.
- N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, "Quantifying memorization across neu- ral language models," in ICLR, 2022.
- B. Biggio, B. Nelson, and P. Laskov, "Poisoning attacks against support vector machines," in ICML, 2012.
- T. Gu, B. Dolan-Gavitt, and S. Garg, "Badnets: Iden- tifying vulnerabilities in the machine learning model supply chain," IEEE Access, 2017.
- X. Chen, C. Liu, B. Li, K. Lu, and D. Song, "Targeted backdoor attacks on deep learning systems using data poisoning," arXiv, 2017.
- A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumitras, and T. Goldstein, "Poison frogs! targeted clean-label poisoning attacks on neural net- works," in NeurIPS, 2018.
- E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, "How to backdoor federated learning," in AISTATS, 2020.
- M. Fang, X. Cao, J. Jia, and N. Gong, "Local model poi- soning attacks to byzantine-robust federated learning," in USENIX Security Symposium, 2020.
- A. Kerckhoffs, "La cryptographie militaire," Journal des sciences militaires, 1883.
- N. Carlini, "Poisoning the unlabeled dataset of semi- supervised learning," in USENIX Security, 2021.
- H. Liu, J. Jia, and N. Z. Gong, "Poisonedencoder: Poi- soning the unlabeled pre-training data in contrastive learning," in USENIX Security Symposium, 2022.
- Y. Huang, S. Gupta, M. Xia, K. Li, and D. Chen, "Catas- trophic jailbreak of open-source llms via exploiting generation," arXiv, 2023.
- Y. Du, A. Bosselut, and C. D. Manning, "Synthetic disinformation attacks on automated fact verification systems," in AAAI, 2022.
- O. Yoran, T. Wolfson, O. Ram, and J. Berant, "Making retrieval-augmented language models robust to irrele- vant context," in ICLR, 2023.
- T. Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez, "Raft: Adapting language model to domain specific rag," arXiv, 2024.
- J. Steinhardt, P. W. W. Koh, and P. S. Liang, "Certified defenses for data poisoning attacks," NeurIPS, 2017.
- J. Jia, Y. Liu, Y. Hu, and N. Z. Gong, "Pore: Provably robust recommender systems against data poisoning attacks," in USENIX Security Symposium, 2023.
- S. Shan, A. N. Bhagoji, H. Zheng, and B. Y. Zhao, "Poison forensics: Traceback of data poisoning attacks in neural networks," in USENIX Security Symposium, 2022.
- F. Jelinek, "Interpolated estimation of markov source parameters from sparse data," in Proc. Workshop on Pattern Recognition in Practice, 1980.
- E. Kortukov, A. Rubinstein, E. Nguyen, and S. J. Oh, "Studying large language model behaviors under realis- tic knowledge conflicts," arXiv, 2024.
Cited by
Sorted by citations
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671470 · Cited by 827 · see citing works
Ding, W., Abdel‐Basset, M., Ali, A. M., & Moustafa, N. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112663 · Cited by 41 · see citing works
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28 · see citing works
“Other work on retrieval corruption and federated training manipulation, studies scenarios where adversaries distort contextual inputs in retrieval-augmented generation pipelines [117] or inject malicious updates in federated learning settings to corrupt distributed training data [118].”
RAGVA: Engineering retrieval augmented generation-based virtual assistants in practice
Yang, R., Fu, M. C., Tantithamthavorn, C. K., Arora, C., Vandenhurk, L., & Chua, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112436 · Cited by 34 · see citing works
(2025) · doi:10.1145/3833415 · Cited by 18 · see citing works
Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generation
Li, Y., Liu, G., Wang, C., & Yang, Y. (2024) · arXiv · doi:10.48550/arxiv.2406.19234 · Cited by 17 · see citing works
“We use two datasets commonly employed in RAG attack research: Natural Questions [24] and Trivia-QA [25], in accordance with existing works [7], [26], [27].”
Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generation
Li, Y., Liu, G., Wang, C., & Yang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889013 · Cited by 31 · see citing works
AI versus AI in Financial Crimes & Detection: GenAI Crime Waves to Co-Evolutionary AI
Kurshan, E., Mehta, D. B., & Balch, T. R. (2024) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3677052.3698655 · Cited by 22 · see citing works
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Chen, Z., Xiang, Z., Xiao, C., Song, D., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.12784 · Cited by 10 · see citing works
Securing Retrieval-Augmented Generation Pipelines: A Comprehensive Framework
Nandagopal, S. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.1.2 · Cited by 7 · see citing works
Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
Liu, Q., Mo, W., Tong, T., Xu, J., Wang, F., Xiao, C., & Chen, M. (2024) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton63246.2024.10735305 · Cited by 6 · see citing works
Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
Li, A., Zhou, Y., Raghuram, V. C., Goldstein, T., & Goldblum, M. (2025) · arXiv · doi:10.48550/arxiv.2502.08586 · Cited by 2 · see citing works
AI Versus AI in Financial Crimes & Detection: GenAI Crime Waves to Co-Evolutionary AIPreprint
Kurshan, E., Mehta, D. B., Balch, T. R., & Bruss, B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4971863 · Cited by 2 · see citing works
Enhancing Privacy And Security In Rag-Based Generative AI ApplicationsPreprint
Mukherjee, S., & Panda, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5162147 · Cited by 2 · see citing works
Memory Injection Attacks on LLM Agents via Query-Only Interaction
Dong, S., Xu, S., He, P., Li, Y., Tang, J., Liu, T., Liu, H., & Xiang, Z. (2025) · arXiv · doi:10.48550/arxiv.2503.03704 · Cited by 1 · see citing works
“…such as the attacker having privileged access to directly manipulate the agent’s memory or inject triggers into other users’ queries [5, 55, 37], we consider a more challenging and realistic threat model with the following constraints: 1) The attacker behaves like a regular user and…”
Adversarial Search Engine Optimization for Large Language Models
Nestaas, F., Debenedetti, E., & Tramèr, F. (2024) · arXiv · doi:10.48550/arxiv.2406.18382 · Cited by 1 · see citing works
“Prior [34] and concurrent works [4, 22, 33, 34] study attacks against RAG systems, where an LLM can query an external database.”
BackgroundBackdoor threats in large language models—a survey
Liu, S., Pan, Y. F., Hong, K., Fei, R., Lin, C., Li, Q., & Shen, C. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4351-3 · Cited by 1 · see citing works
Chen, X., He, B., Lin, H., Han, X., Wang, T., Cao, B., Sun, L., & Sun, Y. (2026) · ACM Transactions on Information Systems · doi:10.1145/3788865 · Cited by 1 · see citing works
Push and Pull : Defending against Retrieval Poisoning Attacks via Embedding Space Reshaping
He, L., Zhang, X., Liu, Q., Li, C., Liu, Z., Zhou, P., & Su, S. (2026) · ACM Transactions on Information Systems · doi:10.1145/3800948 · Cited by 1 · see citing works
Dissecting Adversarial Robustness of Multimodal LM Agents
Wu, C. H., Shah, R., Koh, J. Y., Salakhutdinov, R., Fried, D., & Raghunathan, A. (2024) · arXiv · doi:10.48550/arxiv.2406.12814 · Cited by 1 · see citing works
Louck, Y. (2026) · arXiv · doi:10.48550/arxiv.2606.24322
“…poisoning that re-emerges across unrelated sessions [5] (echoing the training-time sleeper-agent threat [6]), memory control-flow attacks that hijack tool selection [7], retrieval corruption [8], memory injection [9], and agent backdoors [10], [11], with reported injection success up to ∼ 99 .”
MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval
Srivastava, S. S., & He, H. (2025) · arXiv · doi:10.48550/arxiv.2512.16962
“PoisonedRAG[10] formulates knowledge corruption as an optimization problem and shows that injecting only a handful of adversarial texts into the knowledge database can steer the model towards an attacker-chosen answer with success rates approaching 90%.”
Hidden in Memory: Sleeper Memory Poisoning in LLM Agents
Pulipaka, S., Hlebik, S., Raghav, L., Abdelnabi, S., Raina, V., Sheth, I., & Fritz, M. (2026) · arXiv · doi:10.48550/arxiv.2605.15338
MemMorph: Tool Hijacking in LLM Agents via Memory Poisoning
Zhang, X., Zheng, Y., Xu, Z., Zhou, K., Shen, B., Ou, H., Zhang, T., & Lam, K.-Y. (2026) · arXiv · doi:10.48550/arxiv.2605.26154
MAPLE-Guard: Memory-Aware Link Enforcement Against Memory-Link Poisoning in Multi-Agent Systems
Xiong, W., Zhou, Y., Wang, J., Gu, S., Tang, B., Li, Z., Xiong, F., Wen, Y., & Wen, M. (2026) · arXiv · doi:10.48550/arxiv.2608.00426
DeRAG: Decentralized Multi-Source RAG System with Optimized Pyth Network
Yu, J., & Sato, H. (2024) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa63168.2024.00022
Kurshan, E., Mehta, D. B., Balch, T. R., Byrd, D., & Shen, H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5377717
Kurshan, E., Balch, T. R., Mehta, D. B., & Byrd, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5385017
Beshane, N. K. S. (2026) · International Journal of Computational and Experimental Science and Engineering · doi:10.5281/zenodo.18364919
Prompt Injection Attack to Tool Selection in LLM Agents
Shi, J., Yuan, Z., Tie, G., Zhou, P., Gong, N. Z., & Sun, L. (2025) · arXiv · doi:10.48550/arxiv.2504.19793 · Cited by 2 · see citing works
- Authors
- Zou, W., Geng, R., Wang, B., & Jia, J.
- Affiliations
- Pennsylvania State University (US)
- Illinois Institute of Technology (US)
- ORCID
- Wang, Binghui
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.07867
- OpenAlex
- W4391801218
- Cited by
- 16
- References
- 0
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- retrieval-augmented generation, large language models, adversarial attacks, attack defense, hallucinations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
