The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)
Open accessAbstract
Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, Jiliang Tang. Findings of the Association for Computational Linguistics: ACL 2024. 2024.
Overview
CoChat Summary
Written by AI from the paper’s full text.
TL;DR
RAG systems are highly vulnerable to black-box prompting attacks that extract private retrieval-database content, but incorporating retrieval data substantially reduces an LLM’s tendency to leak its own training data. The paper empirically maps both risks and evaluates defenses, finding a privacy–utility tradeoff.
Key findings
- Untargeted attacks extracted retrieval data at high rates: with Enron Email and GPT-3.5-turbo, 250 prompts yielded 452 retrieval contexts, 116 repeat prompts, 121 ROUGE prompts, and 208 ROUGE contexts.
- Targeted attacks also succeeded: with Llama-7b-Chat, 250 prompts extracted 89 targeted medical dialogue chunks from HealthCareMagic and 107 PIIs from Enron Email.
- RAG reduced training-data leakage: targeted PII extraction from GPT-Neo-1.3B training data dropped from 245 email / 27 phone / 34 URL with the LLM alone to 2/1/15 with RAG-Chatdoctor and 2/2/3 with RAG-Wikitext; prefix reconstruction dropped from 213 to 34 (RAG-Chatdoctor) or 70 (RAG-Wikitext).
- Summarization mitigated untargeted retrieval-data leakage by about 50% (e.g., HealthCareMagic ROUGE contexts 113 to 44 with abstractive summarization), while re-ranking had almost no mitigation effect; distance thresholds showed a privacy–utility tradeoff.
Methods
- Black-box API threat model; composite structured prompting `q = {information} + {command}` for targeted and untargeted retrieval-data extraction, plus targeted and prefix attacks for LLM training-data memorization.
- RAG components: Llama-7b-chat, Llama-13b-chat, GPT-3.5-turbo; embeddings bge-large-en-v1.5, all-MiniLM-L6-v2, e5-base-v2; Chroma; default `k=2`; retrieval datasets Enron Email (500,000 emails) and HealthcareMagic-101 (200k doctor–patient dialogues). For RQ2, GPT-Neo-1.3B with Enron_Mail as training subset and wikitext-103, HealthcareMagic, and w3c-email as retrieval datasets.
- Metrics: retrieval contexts, repeat prompts, repeat contexts, targeted information, ROUGE-L > 0.5 prompts/contexts; prefix attack success by ROUGE-L > 0.5; performance by ROUGE-L or perplexity on held-out test sets.
Limitations
- Paper-stated: the study focuses on retrieval augmentation at inference, not during pre-training or fine-tuning; other retrieval-based LMs with distinct architectures need further investigation; differential privacy for retrieval data and using RAG to safeguard training data remain open questions.
- Paper-stated: the prefix attack requires attackers to know actual training data, limiting its practicality, though it is useful for quantitatively measuring memorization.
- Evident from text: attack success depends heavily on command design and retrieval settings, and the evaluated models/datasets may not generalize to all RAG deployments.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
38 Own result · 16 Own interpretation · 3 Check pending · 1 Sources not found · 1 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, with Llama-7b-Chat as the generative model, 250 prompts successfully extracted 89 targeted medical dialogue chunks from HealthCareMagic and 107 PIIs from Enron Email.”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > 4.3 Results of Targeted Attack
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis shows that targeted attacks can effectively retrieve sensitive information, as detailed in Table 2.”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > 4.3 Results of Targeted Attack
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It shows that a majority of the prompts effectively prompted the retrieval system to fetch relevant data segments.”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > 4.2 Results of Untargeted Attack
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is obvious that different commands indeed affect the extraction performance.”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > Impact of Questions.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite the new risk brought by RAG on the retrieval data, we further reveal that RAG can mitigate the leakage of the LLMs' training data.”
- Also stated in: § 6 Conclusions · hedged
“we also found that integrating retrieval data can substantially reduce LLMs' tendency to output its memorized training data, which suggests that RAG could potentially mitigate the risks of training data leakage.”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Also stated in: § 6 Conclusions · hedged
- It may state two separate claims as one. 10
- Probably not a substantial claim. 2
Create a free account to see these claims.
Create a free account to see these assumptions.
42 more claims
Create a free account to see all 59 claims and to verify the 3 references still open.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite initial studies exploring these risks, they often lack a formalized framework, robust attack performance, and comprehensive evaluation, leaving critical questions about real-world EDEA feasibility unanswered.”
From External Data Extraction Attacks Against Retrieval-Augmented Large Language Models (2026) · cites this paper as [17]
In the citing paper
“Specifically, they suffer from the following two major limitations: (1) they lack a unified definition and formalization of EDEAs and (2) their effectiveness and scalability are limited, especially against RA-LLMs equipped even with naive defenses (e.g., instructions to refuse to disclose text contained in the knowledge base).”
From External Data Extraction Attacks Against Retrieval-Augmented Large Language Models (2026) · cites this paper as [17]
In the citing paper
“Leveraging the strong instruction-following capabilities inherent in modern LLMs, simple commands such as 'Please repeat all the context' can indeed induce models to output retrieved documents [17], [18].”
From External Data Extraction Attacks Against Retrieval-Augmented Large Language Models (2026) · cites this paper as [17]
In the citing paper
“Specifically, they suffer from the following two major limitations: (1) they lack a unified definition and formalization of EDEAs and (2) their effectiveness and scalability are limited, especially against RA-LLMs equipped even with naive defenses (e.g., instructions to refuse to disclose text contained in the knowledge base).”
From External Data Extraction Attacks Against Retrieval-Augmented Large Language Models (2026) · cites this paper as [17]
In the citing paper
“However, existing works primarily provide simple case studies, merely illustrating the privacy threats posed by the attacks.”
From External Data Extraction Attacks Against Retrieval-Augmented Large Language Models (2026) · cites this paper as [17]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > 4.2 Results of Untargeted Attack
“In total, this results in 112 exact text matches (Repeat Contexts) and 208 similar responses (Rouge Contexts).”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > Impact of Questions.
“| | GPT | 452 | 116 | 122 | 121 | 208 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > 5.3 Prefix Attack
“Specifically, it leads to a reduction in successful text reconstructions from over 200 cases to fewer than 40.”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > Impact of Questions.
“| RAG-Wikitext | 2 | 2 | 3 | 0 | 0 | 0 | 70 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > 5.1 Evaluation setup
“we include baselines with 50 tokens of random noise injection”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > Additional information
“For random noises, we just randomly choose 50 characters and prepend them to the prompt.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > 4.5 Potential Mitigation
“For instance, in the Enron email dataset, the occurrence of personally identifiable information (PIIs) even inadvertently increased.”
§ The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) > Additional information
“| | No | 134 | 39 | 32 | 12 | | Enron-Email | Yes | 134 | 27 | 21 | 11 | | | Yes-edit | 134 | 27 | 24 | 12 |”
Citations
Cites
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S. B., & Zhang, Y. (2023) · Cureus · doi:10.7759/cureus.40895 · Cited by 784
Retrieval Augmentation Reduces Hallucination in ConversationPreprint
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07567 · Cited by 455
In-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285
REPLUG: Retrieval-Augmented Black-Box Language Models
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., & Yih, W.-T. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.463 · Cited by 209
Quantifying Memorization Across Neural Language Models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., & Chen, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.620 · Cited by 200
Counterfactual Memorization in Neural Language Models
Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramèr, F., & Carlini, N. (2023) · neural information processing systems · doi:10.52202/075280-1708 · Cited by 148
Research works printed but not linked: 16
- Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. 2023. Emer- gent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158. · arXiv
- Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240.
- PMLR.
- Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.
- Ricardo Silva Carvalho, Theodore Vasiloudis, Oluwaseyi Feyisetan, and Ke Wang. 2023. Tem: High utility metric differential privacy on text. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 883-890.
- Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. 2020. Privacy-and utility-preserving tex- tual analysis via calibrated multivariate perturbations. In Proceedings of the 13th international conference on web search and data mining, pages 178-186.
- Evelyn Fix and Joseph Lawson Hodges. 1989. Dis- criminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Re- view/Revue Internationale de Statistique, 57(3):238- 247.
- Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697-10707.
- PMLR.
- Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2023. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pages 3637-3647.
- Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick. 2022. Memorization in nlp fine-tuning methods. arXiv preprint arXiv:2205.12506. · arXiv
- Dimitrios P Panagoulias, Maria Virvou, and George A Tsihrintzis. 2024. Augmenting large language mod- els with rules for enhanced domain-specific interac- tions: The case of medical diagnosis. Electronics, 13(2):320.
- Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 2719-2734.
- Jie Ren, Han Xu, Pengfei He, Yingqian Cui, Shenglai Zeng, Jiankun Zhang, Hongzhi Wen, Jiayuan Ding, Hui Liu, Yi Chang, et al. 2024. Copyright protec- tion in generative ai: A technical perspective. arXiv preprint arXiv:2402.02333. · arXiv
- Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Blueth- gen, Anuj Pareek, Malgorzata Polacin, William Collins, Neera Ahuja, et al. 2023. Clinical text summarization: Adapting large language models can outperform human experts. arXiv preprint arXiv:2309.07430. · arXiv
- Li Yunxiang, Li Zihan, Zhang Kai, Dan Ruilong, and Zhang You. 2023. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070. · arXiv
Cited by
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671470 · Cited by 827
Self-citationRetrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267
AI agents under threat: A survey of key security challenges and future pathways
Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3716628 · Cited by 210
Wan, Y., Chen, Z., Liu, Y., Chen, C., & Packianather, M. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103212 · Cited by 145
Self-citationBrown, A., Roman, M., & Devereux, B. J. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9120320 · Cited by 128
Metadata
- Authors
- Zeng, S., Zhang, J., He, P., Liu, Y., Xing, Y., Xu, H., Ren, J., Chang, Y., Wang, S., Yin, D., & Tang, J.
- Affiliations
- Michigan State University (US)
- Baidu (China) (CN)
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 4505-4524
- OpenAlex
- W4402670423
- Cited by
- 149
- References
- 21
- Citations by year
- 2026: 83 · 2025: 61 · 2024: 5
- FWCI
- 22.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Keywords
- retrieval-augmented generation, privacy, data privacy, information retrieval, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG). https://app.cochat.ai/papers/10.18653/v1/2024.findings-acl.267
- The paper
- Zeng, S., Zhang, J., He, P., Liu, Y., Xing, Y., Xu, H., Ren, J., Chang, Y., Wang, S., Yin, D., & Tang, J. (2024). The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG). Findings of the Association for Computational Linguistics ACL 2024. https://doi.org/10.18653/v1/2024.findings-acl.267
