Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 152Privacy-Preserving Technologies in DataTopic ModelingPrivacy, Security, and Data Protection

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.

The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)

Open access
Zeng, Shenglai, Zhang, Jiankun, He, Pengfei, Liu, Yiding, Xing, Yue, Xu, Han, Ren, Jie, Chang, Yi, Wang, Shuaiqiang, Yin, Dawei, Tang, Jiliang·2024·Findings of the Association for Computational Linguistics ACL 2024

Abstract

Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, Jiliang Tang. Findings of the Association for Computational Linguistics: ACL 2024. 2024.

Overview

CoChat Summary

Written by AI from the paper’s full text.

TL;DR

RAG systems are highly vulnerable to black-box prompting attacks that extract private retrieval-database content, but incorporating retrieval data substantially reduces an LLM’s tendency to leak its own training data. The paper empirically maps both risks and evaluates defenses, finding a privacy–utility tradeoff.

Key findings

  • Untargeted attacks extracted retrieval data at high rates: with Enron Email and GPT-3.5-turbo, 250 prompts yielded 452 retrieval contexts, 116 repeat prompts, 121 ROUGE prompts, and 208 ROUGE contexts.
  • Targeted attacks also succeeded: with Llama-7b-Chat, 250 prompts extracted 89 targeted medical dialogue chunks from HealthCareMagic and 107 PIIs from Enron Email.
  • RAG reduced training-data leakage: targeted PII extraction from GPT-Neo-1.3B training data dropped from 245 email / 27 phone / 34 URL with the LLM alone to 2/1/15 with RAG-Chatdoctor and 2/2/3 with RAG-Wikitext; prefix reconstruction dropped from 213 to 34 (RAG-Chatdoctor) or 70 (RAG-Wikitext).
  • Summarization mitigated untargeted retrieval-data leakage by about 50% (e.g., HealthCareMagic ROUGE contexts 113 to 44 with abstractive summarization), while re-ranking had almost no mitigation effect; distance thresholds showed a privacy–utility tradeoff.

Methods

  • Black-box API threat model; composite structured prompting `q = {information} + {command}` for targeted and untargeted retrieval-data extraction, plus targeted and prefix attacks for LLM training-data memorization.
  • RAG components: Llama-7b-chat, Llama-13b-chat, GPT-3.5-turbo; embeddings bge-large-en-v1.5, all-MiniLM-L6-v2, e5-base-v2; Chroma; default `k=2`; retrieval datasets Enron Email (500,000 emails) and HealthcareMagic-101 (200k doctor–patient dialogues). For RQ2, GPT-Neo-1.3B with Enron_Mail as training subset and wikitext-103, HealthcareMagic, and w3c-email as retrieval datasets.
  • Metrics: retrieval contexts, repeat prompts, repeat contexts, targeted information, ROUGE-L > 0.5 prompts/contexts; prefix attack success by ROUGE-L > 0.5; performance by ROUGE-L or perplexity on held-out test sets.

Limitations

  • Paper-stated: the study focuses on retrieval augmentation at inference, not during pre-training or fine-tuning; other retrieval-based LMs with distinct architectures need further investigation; differential privacy for retrieval data and using RAG to safeguard training data remain open questions.
  • Paper-stated: the prefix attack requires attackers to know actual training data, limiting its practicality, though it is useful for quantitatively measuring memorization.
  • Evident from text: attack success depends heavily on command design and retrieval settings, and the evaluated models/datasets may not generalize to all RAG deployments.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2024.findings-acl.267

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 The RAG system and potential risks.
Figure 1 The RAG system and potential risks.CC-BY · From the PDF
Figure 2 Ablation study on command part. (R) means Repeat Contexts and (RG) means Rouge Contexts
Figure 2 Ablation study on command part. (R) means Repeat Contexts and (RG) means Rouge ContextsCC-BY · From the PDF
Figure 3 Ablation study on number of retrieved docs per query k.
Figure 3 Ablation study on number of retrieved docs per query k.CC-BY · From the PDF
Figure 4 Potential post-processing mitigation strategies. The impact of reranking on (a) targeted attacks,(b) untargetted attacks; and the impact of summarization on (c) untargeted attacks and (d) targeted attacks
Figure 4 Potential post-processing mitigation strategies. The impact of reranking on (a) targeted attacks,(b) untargetted attacks; and the impact of summarization on (c) untargeted attacks and (d) targeted attacksCC-BY · From the PDF
Figure 5 The impact of retrieval threshold on performance and privacy leakage
Figure 5 The impact of retrieval threshold on performance and privacy leakageCC-BY · From the PDF
Figure 6 Ablation study on embedding models.
Figure 6 Ablation study on embedding models.CC-BY · From the PDF

Verification

Claims the paper makes

38 Own result · 16 Own interpretation · 3 Check pending · 1 Sources not found · 1 No basis given

  • It may state two separate claims as one. 10
  • Probably not a substantial claim. 2

Create a free account to see these claims.

Create a free account to see these assumptions.

42 more claims

Create a free account to see all 59 claims and to verify the 3 references still open.

See all claims — free account

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 21 references

Research works printed but not linked: 16

  1. Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. 2023. Emer- gent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158. · arXiv
  2. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240.
  3. PMLR.
  4. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.
  5. Ricardo Silva Carvalho, Theodore Vasiloudis, Oluwaseyi Feyisetan, and Ke Wang. 2023. Tem: High utility metric differential privacy on text. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 883-890.
  6. Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. 2020. Privacy-and utility-preserving tex- tual analysis via calibrated multivariate perturbations. In Proceedings of the 13th international conference on web search and data mining, pages 178-186.
  7. Evelyn Fix and Joseph Lawson Hodges. 1989. Dis- criminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Re- view/Revue Internationale de Statistique, 57(3):238- 247.
  8. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697-10707.
  9. PMLR.
  10. Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2023. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pages 3637-3647.
  11. Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick. 2022. Memorization in nlp fine-tuning methods. arXiv preprint arXiv:2205.12506. · arXiv
  12. Dimitrios P Panagoulias, Maria Virvou, and George A Tsihrintzis. 2024. Augmenting large language mod- els with rules for enhanced domain-specific interac- tions: The case of medical diagnosis. Electronics, 13(2):320.
  13. Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 2719-2734.
  14. Jie Ren, Han Xu, Pengfei He, Yingqian Cui, Shenglai Zeng, Jiankun Zhang, Hongzhi Wen, Jiayuan Ding, Hui Liu, Yi Chang, et al. 2024. Copyright protec- tion in generative ai: A technical perspective. arXiv preprint arXiv:2402.02333. · arXiv
  15. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Blueth- gen, Anuj Pareek, Malgorzata Polacin, William Collins, Neera Ahuja, et al. 2023. Clinical text summarization: Adapting large language models can outperform human experts. arXiv preprint arXiv:2309.07430. · arXiv
  16. Li Yunxiang, Li Zihan, Zhang Kai, Dan Ruilong, and Zhang You. 2023. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070. · arXiv

Cited by

Items 101–152 of 152

Metadata

Authors
Zeng, S., Zhang, J., He, P., Liu, Y., Xing, Y., Xu, H., Ren, J., Chang, Y., Wang, S., Yin, D., & Tang, J.
Affiliations
  • Michigan State University (US)
  • Baidu (China) (CN)
Venue
Findings of the Association for Computational Linguistics ACL 2024
Published
2024
Pages
4505-4524
OpenAlex
W4402670423
Cited by
149
References
21
Citations by year
2026: 83 · 2025: 61 · 2024: 5
FWCI
22.45
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
Keywords
retrieval-augmented generation, privacy, data privacy, information retrieval, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review