CoChat
Sign Up
PreprintCited by 10Privacy-Preserving Technologies in DataAdversarial Robustness in Machine LearningPrivacy, Security, and Data Protection

Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey

Open access
Smith, Victoria, Shamsabadi, Ali Shahin, Ashurst, Carolyn, Weller, Adrian·2023·arXiv
DOI 10.48550/arxiv.2310.01424arXiv 2310.01424
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have shown greatly enhanced performance in recent years, attributed to increased size and extensive training data. This advancement has led to widespread interest and adoption across industries and the public. However, training data memorization in Machine Learning models scales with model size, particularly concerning for LLMs. Memorized text sequences have the potential to be directly leaked from LLMs, posing a serious threat to data privacy. Various techniques have been developed to attack LLMs and extract their training data. As these models continue to grow, this issue becomes increasingly critical. To help researchers and policymakers understand the state of knowledge around privacy attacks and mitigations, including where more work is needed, we present the first SoK on data privacy for LLMs. We (i) identify a taxonomy of salient dimensions where attacks differ on LLMs, (ii) systematize existing attacks, using our taxonomy of dimensions to highlight key trends, (iii) survey existing mitigation strategies, highlighting their strengths and limitations, and (iv) identify key gaps, demonstrating open problems and areas for concern.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.01424

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Outlying data are especially at risk of being memorized and leaked, which in the case of personal data may lead to privacy disclosure.

    “Outlying data are especially at risk of being memorized and leaked, which in the case of personal data may lead to privacy disclosure (Smith et al 2023).”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Smith et al 2023

    Section 3, “Training Data Memorization in LLMs” (page unavailable), says memorized training data can be extracted and that personal information can be among it. Section 7, “Data Extraction Attacks on LLMs” (page unavailable), says such extraction can cause privacy violations. The copy does not support the claim that outlying data are especially at risk; it instead identifies duplicated or highly repeated sequences as more likely to be memorized and leaked. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • The success of unlearning depends on the specific target data and the domain of the to-be-forgotten data.

    “Furthermore, the success of unlearning depends on the specific target data and the domain of the to-be-forgotten data (Smith et al 2023).”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Smith et al 2023

    Section 12.2 (Machine Unlearning; page unavailable) says the forgetting threshold depends on which data samples are chosen and reports that larger forget sets degraded model performance. This lends limited support to the importance of target data, but the passage concerns the measurement of forgetting rather than directly establishing that unlearning success depends on the target samples. It does not address whether success depends on the domain of the data to be forgotten. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 131

  1. X. Yang, A. Chen, N. PourNejatian, H. C. Shin, K. E. Smith, C. Parisien, C. Compas, C. Martin, A. B. Costa, M. G. Flores et al., "A large language model for electronic health records," npj Digital Medicine, vol. 5, no. 1, p. 194, 2022.
  2. G. Benegas, S. S. Batra, and Y. S. Song, "Dna language models are powerful zero-shot predictors of non-coding variant effects," bioRxiv, pp. 2022-08, 2022.
  3. H.-L. Li, Y.-H. Pang, and B. Liu, "Bioseq-blm: a platform for ana- lyzing dna, rna and protein sequences based on biological language models," Nucleic acids research, vol. 49, no. 22, pp. e129-e129, 2021.
  4. S. Shaghaghian, L. Y. Feng, B. Jafarpour, and N. Pogrebnyakov, "Customizing contextualized language models for legal document reviews," in 2020 IEEE International Conference on Big Data (Big Data). IEEE, 2020, pp. 2139-2148.
  5. I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. An- droutsopoulos, "Legal-bert:" preparing the muppets for court'"." in EMNLP (Findings), 2020, pp. 2898-2904.
  6. R. Shokri, M. Stronati, C. Song, and V. Shmatikov, "Membership inference attacks against machine learning models," in 2017 IEEE symposium on security and privacy (SP). IEEE, 2017, pp. 3-18.
  7. S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, "Privacy risk in machine learning: Analyzing the connection to overfitting," in 2018 IEEE 31st computer security foundations symposium (CSF). IEEE, 2018, pp. 268-282.
  8. L. Song and P. Mittal, "Systematic evaluation of privacy risks of machine learning models," in 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 2615-2632.
  9. A. Salem, G. Cherubin, D. Evans, B. Köpf, A. Paverd, A. Suri, S. Tople, and S. Zanella-Béguelin, "Sok: Let the privacy games begin! a unified treatment of data inference privacy in machine learning," in 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023, pp. 327-345.
  10. M. Rigaki and S. Garcia, "A survey of privacy attacks in machine learning," ACM Computing Surveys, vol. 56, no. 4, pp. 1-34, 2023.
  11. H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr, "What does it mean for a language model to preserve privacy?" in 2022 ACM Conference on Fairness, Accountability, and Trans- parency, 2022, pp. 2280-2292.
  12. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler et al., "Emergent abil- ities of large language models," arXiv preprint arXiv:2206.07682, 2022. · arXiv
  13. N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song, "The secret sharer: Evaluating and testing unintended memorization in neural networks." in USENIX Security Symposium, vol. 267, 2019.
  14. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., "Ex- tracting training data from large language models," in 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 2633-2650.
  15. A. Thomas, D. I. Adelani, A. Davody, A. Mogadala, and D. Klakow, "Investigating the impact of pre-trained word embeddings on mem- orization in neural networks," in Text, Speech, and Dialogue: 23rd International Conference, TSD 2020, Brno, Czech Republic, Septem- ber 8-11, 2020, Proceedings 23. Springer, 2020, pp. 273-281.
  16. S. Ramaswamy, O. Thakkar, R. Mathews, G. Andrew, H. B. McMa- han, and F. Beaufays, "Training production language models without memorizing user data," arXiv preprint arXiv:2009.10031, 2020. · arXiv
  17. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, "Quantifying memorization across neural language mod- els," arXiv preprint arXiv:2202.07646, 2022. · arXiv
  18. R. T. McCoy, P. Smolensky, T. Linzen, J. Gao, and A. Celikyilmaz, "How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven," arXiv preprint arXiv:2111.09509, 2021. · arXiv
  19. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhari- wal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Lan- guage models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  20. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., "Chain-of-thought prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 24 824-24 837, 2022.
  21. S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson et al., "Language models (mostly) know what they know," arXiv preprint arXiv:2207.05221, 2022. · arXiv
  22. R. Staab, M. Vero, M. Balunović, and M. Vechev, "Beyond mem- orization: Violating privacy via inference with large language mod- els," arXiv preprint arXiv:2310.07298, 2023. · arXiv
  23. N. Kshetri, "Cybercrime and privacy threats of large language models," IT Professional, vol. 25, no. 3, pp. 9-13, 2023.
  24. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., "On the opportunities and risks of foundation models," arXiv preprint arXiv:2108.07258, 2021. · arXiv
  25. H. Wang, M. Tan, M. Yu, S. Chang, D. Wang, K. Xu, X. Guo, and S. Potdar, "Extracting multiple-relations in one-pass with pre-trained transformers," arXiv preprint arXiv:1902.01030, 2019. · arXiv
  26. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
  27. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter," arXiv preprint arXiv:1910.01108, 2019. · arXiv
  28. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized bert pretraining approach," arXiv preprint arXiv:1907.11692, 2019. · arXiv
  29. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  30. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang et al., "Gpt-neox-20b: An open-source autoregressive language model," arXiv preprint arXiv:2204.06745, 2022. · arXiv
  31. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
  32. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., "Opt: Open pre-trained transformer language models," arXiv preprint arXiv:2205.01068, 2022. · arXiv
  33. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485-5551, 2020.
  34. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, "Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension," arXiv preprint arXiv:1910.13461, 2019. · arXiv
  35. J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groen- eveld, M. Mitchell, and M. Gardner, "Documenting large webtext corpora: A case study on the colossal clean crawled corpus," arXiv preprint arXiv:2104.08758, 2021. · arXiv
  36. S. M. Xie, A. Raghunathan, P. Liang, and T. Ma, "An explanation of in-context learning as implicit bayesian inference," arXiv preprint arXiv:2111.02080, 2021. · arXiv
  37. S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, "Rethinking the role of demonstrations: What makes in-context learning work?" arXiv preprint arXiv:2202.12837, 2022. · arXiv
  38. N. Kandpal, E. Wallace, and C. Raffel, "Deduplicating training data mitigates privacy risks in language models," in International Conference on Machine Learning. PMLR, 2022, pp. 10 697-10 707.
  39. K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison- Burch, and N. Carlini, "Deduplicating training data makes language models better," arXiv preprint arXiv:2107.06499, 2021. · arXiv
  40. M. Jagielski, O. Thakkar, F. Tramer, D. Ippolito, K. Lee, N. Carlini, E. Wallace, S. Song, A. Thakurta, N. Papernot et al., "Measur- ing forgetting of memorized training examples," arXiv preprint arXiv:2207.00099, 2022. · arXiv
  41. S. Hisamoto, M. Post, and K. Duh, "Membership inference attacks on sequence-to-sequence models: Is my data in your machine trans- lation system?" Transactions of the Association for Computational Linguistics, vol. 8, pp. 49-63, 2020.
  42. S. Zanella-Béguelin, L. Wutschitz, S. Tople, V. Rühle, A. Paverd, O. Ohrimenko, B. Köpf, and M. Brockschmidt, "Analyzing informa- tion leakage of updates to natural language models," in Proceedings of the 2020 ACM SIGSAC conference on computer and communi- cations security, 2020, pp. 363-375.
  43. A. Jagannatha, B. P. S. Rawat, and H. Yu, "Membership inference attack susceptibility of clinical language models," arXiv preprint arXiv:2104.08305, 2021. · arXiv
  44. C. Chen, X. He, L. Lyu, and F. Wu, "Killing one bird with two stones: Model extraction and attribute inference attacks against bert- based apis," arXiv preprint arXiv:2105.10909, 2021. · arXiv
  45. A. Elmahdy, H. A. Inan, and R. Sim, "Privacy leakage in text classification: A data extraction approach," arXiv preprint arXiv:2206.04591, 2022. · arXiv
  46. Z. Zhang, J. Wen, and M. Huang, "Ethicist: Targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation," arXiv preprint arXiv:2307.04401, 2023. · arXiv
  47. F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri, "Quantifying privacy risks of masked language models using membership inference attacks," arXiv preprint arXiv:2203.03929, 2022. · arXiv
  48. N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, "Membership inference attacks from first principles," in 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022, pp. 1897- 1914.
  49. T. Wu, A. Panda, J. T. Wang, and P. Mittal, "Privacy-preserving in-context learning for large language models," in The Twelfth International Conference on Learning Representations, 2023.
  50. R. Tang, G. Lueck, R. Quispe, H. A. Inan, J. Kulkarni, and X. Hu, "Assessing privacy risks in language models: A case study on summarization tasks," arXiv preprint arXiv:2310.13291, 2023. · arXiv
  51. M. Li, J. Wang, J. Wang, and S. Neel, "Mope: Model perturbation- based privacy attacks on language models," arXiv preprint arXiv:2310.14369, 2023. · arXiv
  52. H. Duan, A. Dziedzic, M. Yaghini, N. Papernot, and F. Boenisch, "On the privacy risk of in-context learning," in The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
  53. W. Fu, H. Wang, C. Gao, G. Liu, Y. Li, and T. Jiang, "Practical mem- bership inference attacks against fine-tuned large language models via self-prompt calibration," arXiv preprint arXiv:2311.06062, 2023. · arXiv
  54. J. Mattern, F. Mireshghallah, Z. Jin, B. Schölkopf, M. Sachan, and T. Berg-Kirkpatrick, "Membership inference attacks against language models via neighbourhood comparison," arXiv preprint arXiv:2305.18462, 2023. · arXiv
  55. M. Meeus, S. Jain, M. Rei, and Y.-A. de Montjoye, "Did the neurons read your book? document-level membership inference for large language models," arXiv preprint arXiv:2310.15007, 2023. · arXiv
  56. W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, "Detecting pretraining data from large language models," arXiv preprint arXiv:2310.16789, 2023. · arXiv
  57. J. Abascal, S. Wu, A. Oprea, and J. Ullman, "Tmi! finetuned models leak private information from their pretraining data," arXiv preprint arXiv:2306.01181, 2023. · arXiv
  58. A. Al-Kaswan, M. Izadi, and A. van Deursen, "Targeted attack on gpt-neo for the satml language model data extraction challenge," arXiv preprint arXiv:2302.07735, 2023. · arXiv
  59. J. X. Morris, W. Zhao, J. T. Chiu, V. Shmatikov, and A. M. Rush, "Language model inversion," arXiv preprint arXiv:2311.13647, 2023. · arXiv
  60. M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ip- polito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee, "Scalable extraction of training data from (production) language models," arXiv preprint arXiv:2311.17035, 2023. · arXiv
  61. W. Yu, T. Pang, Q. Liu, C. Du, B. Kang, Y. Huang, M. Lin, and S. Yan, "Bag of tricks for training data extraction from language models," arXiv preprint arXiv:2302.04460, 2023. · arXiv
  62. A. Al-Kaswan, M. Izadi, and A. van Deursen, "Traces of mem- orisation in large language models for code," arXiv preprint arXiv:2312.11658, 2023. · arXiv
  63. M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, and H. Hajishirzi, "Do membership inference attacks work on large language models?" arXiv preprint arXiv:2402.07841, 2024. · arXiv
  64. E. Tabassi, K. J. Burns, M. Hadjimichael, A. D. Molina-Markham, and J. T. Sexton, "A taxonomy and terminology of adversarial machine learning," NIST IR, vol. 2019, pp. 1-29, 2019.
  65. Y. Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang, H. Tang, C. A. Gunter, and K. Chen, "Understanding membership in- ferences on well-generalized learning models," arXiv preprint arXiv:1802.04889, 2018. · arXiv
  66. L. Watson, C. Guo, G. Cormode, and A. Sablayrolles, "On the im- portance of difficulty calibration in membership inference attacks," arXiv preprint arXiv:2111.08440, 2021. · arXiv
  67. W. Fedus, B. Zoph, and N. Shazeer, "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity," The Journal of Machine Learning Research, vol. 23, no. 1, pp. 5232- 5270, 2022.
  68. F. Mireshghallah, A. Uniyal, T. Wang, D. Evans, and T. Berg- Kirkpatrick, "Memorization in nlp fine-tuning methods," arXiv preprint arXiv:2205.12506, 2022. · arXiv
  69. X. Tang, R. Shin, H. A. Inan, A. Manoel, F. Mireshghallah, Z. Lin, S. Gopi, J. Kulkarni, and R. Sim, "Privacy-preserving in- context learning with differentially private few-shot generation," arXiv preprint arXiv:2309.11765, 2023. · arXiv
  70. H. Duan, A. Dziedzic, N. Papernot, and F. Boenisch, "Flocks of stochastic parrots: Differentially private prompt learning for large language models," Advances in Neural Information Processing Sys- tems, vol. 36, 2024.
  71. S. Biderman, U. S. Prashanth, L. Sutawika, H. Schoelkopf, Q. An- thony, S. Purohit, and E. Raf, "Emergent and predictable memoriza- tion in large language models," arXiv preprint arXiv:2304.11158, 2023. · arXiv
  72. H. A. Inan, O. Ramadan, L. Wutschitz, D. Jones, V. Rühle, J. With- ers, and R. Sim, "Training data leakage analysis in language mod- els," arXiv preprint arXiv:2101.05405, 2021. · arXiv
  73. M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart, "Privacy in pharmacogenetics: An end-to-end case study of per- sonalized warfarin dosing," in 23rd {USENIX} Security Symposium ({USENIX} Security 14), 2014, pp. 17-32.
  74. M. Fredrikson, S. Jha, and T. Ristenpart, "Model inversion attacks that exploit confidence information and basic countermeasures," in Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, 2015, pp. 1322-1333.
  75. E. Lehman, S. Jain, K. Pichotta, Y. Goldberg, and B. C. Wallace, "Does bert pretrained on clinical notes reveal sensitive data?" arXiv preprint arXiv:2104.07762, 2021. · arXiv
  76. P. Basu, T. S. Roy, R. Naidu, Z. Muftuoglu, S. Singh, and F. Mireshghallah, "Benchmarking differential privacy and federated learning for bert models," arXiv preprint arXiv:2106.13973, 2021. · arXiv
  77. S. Hoory, A. Feder, A. Tendler, S. Erell, A. Peled-Cohen, I. Laish, H. Nakhost, U. Stemmer, A. Benjamini, A. Hassidim et al., "Learn- ing and evaluating a differentially private pre-trained language model," in Findings of the Association for Computational Linguis- tics: EMNLP 2021, 2021, pp. 1178-1189.
  78. R. Anil, B. Ghazi, V. Gupta, R. Kumar, and P. Manurangsi, "Large- scale differentially private bert," arXiv preprint arXiv:2108.01624, 2021. · arXiv
  79. D. Yu, H. Zhang, W. Chen, J. Yin, and T.-Y. Liu, "Large scale private learning via low-rank reparametrization," in International Conference on Machine Learning. PMLR, 2021, pp. 12 208-12 218.
  80. C. Dupuy, R. Arava, R. Gupta, and A. Rumshisky, "An efficient dp- sgd mechanism for large scale nlu models," in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 4118-4122.
  81. X. Li, F. Tramer, P. Liang, and T. Hashimoto, "Large language models can be strong differentially private learners," arXiv preprint arXiv:2110.05679, 2021. · arXiv
  82. D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz et al., "Differ- entially private fine-tuning of language models," arXiv preprint arXiv:2110.06500, 2021. · arXiv
  83. F. Mireshghallah, A. Backurs, H. A. Inan, L. Wutschitz, and J. Kulkarni, "Differentially private model compression," arXiv preprint arXiv:2206.01838, 2022. · arXiv
  84. N. Ponomareva, J. Bastings, and S. Vassilvitskii, "Training text- to-text transformers with privacy guarantees," in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 2182-2193.
  85. A. Ginart, L. van der Maaten, J. Zou, and C. Guo, "Submix: Practical private prediction for large-scale language models," arXiv preprint arXiv:2201.00971, 2022. · arXiv
  86. J. Majmudar, C. Dupuy, C. Peris, S. Smaili, R. Gupta, and R. Zemel, "Differentially private decoding in large language models," arXiv preprint arXiv:2205.13621, 2022. · arXiv
  87. J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, "Knowledge unlearning for mitigating privacy risks in language models," arXiv preprint arXiv:2210.01504, 2022. · arXiv
  88. M. Xu, C. Song, Y. Tian, N. Agrawal, F. Granqvist, R. van Dalen, X. Zhang, A. Argueta, S. Han, Y. Deng et al., "Training large- vocabulary neural language models by private federated learning for resource-constrained devices," in ICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
  89. Y. Ishibashi and H. Shimodaira, "Knowledge sanitization of large language models," arXiv preprint arXiv:2309.11852, 2023. · arXiv
  90. X. Wu, J. Li, M. Xu, W. Dong, S. Wu, C. Bian, and D. Xiong, "Depn: Detecting and editing privacy neurons in pretrained language models," arXiv preprint arXiv:2310.20138, 2023. · arXiv
  91. A. Kassem, O. Mahmoud, and S. Saad, "Preserving privacy through dememorization: An unlearning technique for mitigating memo- rization risks in language models," in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 4360-4379.
  92. S. Srivastava, P. Mardziel, Z. Zhang, A. Ahlawat, A. Datta, and J. C. Mitchell, "De-amplifying bias from differential privacy in language model fine-tuning," arXiv preprint arXiv:2402.04489, 2024. · arXiv
  93. T. Aura, T. A. Kuhn, and M. Roe, "Scanning electronic documents for personally identifiable information," in Proceedings of the 5th ACM workshop on Privacy in electronic society, 2006, pp. 41-50.
  94. F. Dernoncourt, J. Y. Lee, O. Uzuner, and P. Szolovits, "De- identification of patient notes with recurrent neural networks," Jour- nal of the American Medical Informatics Association, vol. 24, no. 3, pp. 596-606, 2017.
  95. M. Post, "A call for clarity in reporting bleu scores," arXiv preprint arXiv:1804.08771, 2018. · arXiv
  96. C. Dwork, A. Roth et al., "The algorithmic foundations of differ- ential privacy." Foundations and Trends in Theoretical Computer Science, vol. 9, no. 3-4, pp. 211-407, 2014.
  97. M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, "Deep learning with differential privacy," in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308-318.
  98. S. Song, K. Chaudhuri, and A. D. Sarwate, "Stochastic gradient descent with differentially private updates," in 2013 IEEE global conference on signal and information processing. IEEE, 2013, pp. 245-248.
  99. R. Bassily, A. Smith, and A. Thakurta, "Private empirical risk minimization: Efficient algorithms and tight error bounds," in 2014 IEEE 55th annual symposium on foundations of computer science. IEEE, 2014, pp. 464-473.
  100. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, "Communication-efficient learning of deep networks from decentralized data," in Artificial intelligence and statistics. PMLR, 2017, pp. 1273-1282.
  101. O. Thakkar, S. Ramaswamy, R. Mathews, and F. Beaufays, "Un- derstanding unintended memorization in federated learning," arXiv preprint arXiv:2006.07490, 2020. · arXiv
  102. L. Zhu, Z. Liu, and S. Han, "Deep leakage from gradients," Ad- vances in neural information processing systems, vol. 32, 2019.
  103. J. Deng, Y. Wang, J. Li, C. Shang, H. Liu, S. Rajasekaran, and C. Ding, "Tag: Gradient attack on transformer-based language mod- els," arXiv preprint arXiv:2103.06819, 2021. · arXiv
  104. S. Gupta, Y. Huang, Z. Zhong, T. Gao, K. Li, and D. Chen, "Recovering private text in federated learning of language models," arXiv preprint arXiv:2205.08514, 2022. · arXiv
  105. L. Fowl, J. Geiping, S. Reich, Y. Wen, W. Czaja, M. Goldblum, and T. Goldstein, "Decepticons: Corrupted transformers breach pri- vacy in federated learning for language models," arXiv preprint arXiv:2201.12675, 2022. · arXiv
  106. M. R. U. Rashid, V. A. Dasu, K. Gu, N. Sultana, and S. Mehnaz, "Fltrojan: Privacy leakage attacks against federated language models through selective weight tampering," arXiv preprint arXiv:2310.16152, 2023. · arXiv
  107. J. Li, S. Liu, and Q. Lei, "Beyond gradient and priors in privacy attacks: Leveraging pooler layer inputs of language models in fed- erated learning," arXiv preprint arXiv:2312.05720, 2023. · arXiv
  108. A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage, "Fed- erated learning for mobile keyboard prediction," arXiv preprint arXiv:1811.03604, 2018. · arXiv
  109. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language mod- els," arXiv preprint arXiv:2106.09685, 2021. · arXiv
  110. R. Eldan and M. Russinovich, "Who's harry potter? approximate unlearning in llms," arXiv preprint arXiv:2310.02238, 2023. · arXiv
  111. J. Chen and D. Yang, "Unlearn what you want to forget: Efficient unlearning for llms," arXiv preprint arXiv:2310.20150, 2023. · arXiv
  112. M. Pawelczyk, S. Neel, and H. Lakkaraju, "In-context unlearn- ing: Language models as few shot unlearners," arXiv preprint arXiv:2310.07579, 2023. · arXiv
  113. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, "Bertscore: Evaluating text generation with bert," arXiv preprint arXiv:1904.09675, 2019. · arXiv
  114. N. De Cao, W. Aziz, and I. Titov, "Editing factual knowledge in language models," arXiv preprint arXiv:2104.08164, 2021. · arXiv
  115. K. Meng, D. Bau, A. Andonian, and Y. Belinkov, "Locating and editing factual associations in gpt," Advances in Neural Information Processing Systems, vol. 35, pp. 17 359-17 372, 2022.
  116. Z. Huang, Y. Shen, X. Zhang, J. Zhou, W. Rong, and Z. Xiong, "Transformer-patcher: One mistake worth one neuron," arXiv preprint arXiv:2301.09785, 2023. · arXiv
  117. D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei, "Knowledge neurons in pretrained transformers," arXiv preprint arXiv:2104.08696, 2021. · arXiv
  118. M. Geva, R. Schuster, J. Berant, and O. Levy, "Transformer feed-forward layers are key-value memories," arXiv preprint arXiv:2012.14913, 2020. · arXiv
  119. K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau, "Mass-editing memory in a transformer," arXiv preprint arXiv:2210.07229, 2022. · arXiv
  120. T.-Y. Chang, J. Thomason, and R. Jia, "Do localization meth- ods actually localize memorized data in llms?" arXiv preprint arXiv:2311.09060, 2023. · arXiv
  121. A. Gu and T. Dao, "Mamba: Linear-time sequence modeling with selective state spaces," arXiv preprint arXiv:2312.00752, 2023. · arXiv
  122. A. Priyanshu, S. Vijay, A. Kumar, R. Naidu, and F. Mireshghallah, "Are chatbots ready for privacy-sensitive applications? an investiga- tion into input regurgitation and prompt-induced sanitization," arXiv preprint arXiv:2305.15008, 2023. · arXiv
  123. N. Mireshghallah, H. Kim, X. Zhou, Y. Tsvetkov, M. Sap, R. Shokri, and Y. Choi, "Can llms keep a secret? testing privacy implications of language models via contextual integrity theory," arXiv preprint arXiv:2310.17884, 2023. · arXiv
  124. A. Panda, T. Wu, J. T. Wang, and P. Mittal, "Differentially private in-context learning," arXiv preprint arXiv:2305.01639, 2023. · arXiv
  125. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, "Tree of thoughts: Deliberate problem solving with large language models," arXiv preprint arXiv:2305.10601, 2023. · arXiv
  126. S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, "A survey on multimodal large language models," arXiv preprint arXiv:2306.13549, 2023. · arXiv
  127. P. Hu, Z. Wang, R. Sun, H. Wang, and M. Xue, "M4i: Multi-modal models membership inference," Advances in Neural Information Processing Systems, vol. 35, pp. 1867-1882, 2022.
  128. S. Feng and F. Tramèr, "Privacy backdoors: Stealing data with cor- rupted pretrained models," arXiv preprint arXiv:2404.00473, 2024. · arXiv
  129. S. Zhao, M. Jia, L. A. Tuan, F. Pan, and J. Wen, "Universal vulnerabilities in large language models: Backdoor attacks for in- context learning," arXiv preprint arXiv:2401.05949, 2024. · arXiv
  130. S. Zhao, J. Wen, L. A. Tuan, J. Zhao, and J. Fu, "Prompt as triggers for backdoor attack: Examining the vulnerability in language models," arXiv preprint arXiv:2305.01219, 2023. · arXiv
  131. E. Debenedetti, G. Severi, N. Carlini, C. A. Choquette-Choo, M. Jagielski, M. Nasr, E. Wallace, and F. Tramèr, "Privacy side chan- nels in machine learning systems," arXiv preprint arXiv:2309.05610, 2023. · arXiv

Cited by

Sorted by citations

Authors
Smith, V., Shamsabadi, A. S., Ashurst, C., & Weller, A.
Affiliations
  • The Alan Turing Institute (GB)
  • University of Cambridge (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.01424
arXiv
2310.01424
OpenAlex
W4387355380
Cited by
10
References
0
Citations by year
2026: 2 · 2025: 5 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • UK Research and Innovation (EP/S021612/1)
  • National Institute for Health and Care Research
  • Leverhulme Trust
  • University College London
  • Engineering and Physical Sciences Research Council (EP/V025279/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
Keywords
large language models, data privacy, training data memorization, privacy attacks, attack taxonomy, data leakage

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review