CoChat
Sign Up
PreprintCited by 1Privacy-Preserving Technologies in DataPrivacy, Security, and Data ProtectionAdversarial Robustness in Machine Learning

The DOI matches its DataCite record.

Forgetting Private Textual Sequences in Language Models via Leave-One-Out Ensemble

Open access
Liu, Zhe, Kalinli, Ozlem·2023·arXiv
DOI 10.48550/arxiv.2309.16082arXiv 2309.16082
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent research has shown that language models have a tendency to memorize rare or unique token sequences in the training corpus. After deploying a model, practitioners might be asked to delete any personal information from the model by individuals' requests. Re-training the underlying model every time individuals would like to practice their rights to be forgotten is computationally expensive. We employ a teacher-student framework and propose a novel leave-one-out ensemble method to unlearn the targeted textual sequences that need to be forgotten from the model. In our approach, multiple teachers are trained on disjoint sets; for each targeted sequence to be removed, we exclude the teacher trained on the set containing this sequence and aggregate the predictions from remaining teachers to provide supervision during fine-tuning. Experiments on LibriSpeech and WikiText-103 datasets show that the proposed method achieves superior privacy-utility trade-offs than other counterparts.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2309.16082

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 30

  1. [1][1] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart, 'Model inversion attacks that exploit confidence information and basic countermeasures,' in Proc. ACM SIGSAC , 2015. -
  2. [2][2] Congzheng Song and Vitaly Shmatikov, 'Auditing data provenance in text-generation models,' in Proc. ACM SIGKDD , 2019. -
  3. [3][3] Nicholas Carlini, Chang Liu, ´ Ulfar Erlingsson, Jernej Kos, and Dawn Song, 'The secret sharer: Evaluating and testing unintended memorization in neural networks,' in 28th USENIX Security Symposium , 2019. -
  4. [4][4] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al., 'Extracting training data from large language models,' in 30th USENIX Security Symposium , 2021. -
  5. [5][5] Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang, 'Are large pre-trained language models leaking your personal information?,' arXiv preprint arXiv:2205.12628 , 2022. - · arXiv
  6. [6][6] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, 'Quantifying memorization across neural language models,' arXiv preprint arXiv:2202.07646 , 2022. - · arXiv
  7. [7][7] Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N Sainath, Zhijeng Chen, and Rohit Prabhavalkar, 'An analysis of incorporating an external language model into a sequence-tosequence model,' in Proc. ICASSP , 2018. -
  8. [8][8] Kazuki Irie, Albert Zeyer, Ralf Schl¨ uter, and Hermann Ney, 'Language modeling with deep transformers,' in Proc. Interspeech , 2019. -
  9. [9][9] WRonny Huang, Steve Chien, Om Thakkar, and Rajiv Mathews, 'Detecting unintended memorization in language-modelfused ASR,' in Proc. Interspeech , 2022. -
  10. [10][10] Alessandro Mantelero, 'The EU proposal for a general data protection regulation and the roots of the 'right to be forgotten',' Computer Law & Security Review , vol. 29, no. 3, pp. 229-235, 2013. -
  11. [11][11] Jakub Koneˇ cn` y, H Brendan McMahan, Felix X Yu, Peter Richt´ arik, Ananda Theertha Suresh, and Dave Bacon, 'Federated learning: Strategies for improving communication efficiency,' NeurIPS Workshop on Private Multi-Party Machine Learning , 2016. -
  12. [12][12] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas, 'Communication-efficient learning of deep networks from decentralized data,' in AISTATS , 2017. -
  13. [13][13] Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Franc ¸oise Beaufays, 'Training production language models without memorizing user data,' arXiv preprint arXiv:2009.10031 , 2020. - · arXiv
  14. [14][14] Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays, 'Understanding unintended memorization in language models under federated learning,' in Proc. ACL , 2020. -
  15. [15][15] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith, 'Calibrating noise to sensitivity in private data analysis,' in Theory of Cryptography Conference . Springer, 2006, pp. 265-284. -
  16. [16][16] Cynthia Dwork and Aaron Roth, 'The algorithmic foundations of differential privacy,' Foundations and Trends in Theoretical Computer Science , vol. 9, no. 3-4, pp. 211-407, 2014. -
  17. [17][17] Nicolas Papernot, Mart´ ın Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar, 'Semi-supervised knowledge transfer for deep learning from private training data,' in Proc. ICLR , 2017. -
  18. [18][18] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and ´ Ulfar Erlingsson, 'Scalable private learning with PATE,' in Proc. ICLR , 2018. -
  19. [19][19] Zhe Liu, Xuedong Zhang, and Fuchun Peng, 'Mitigating unintended memorization in language models via alternating teaching,' in Proc. ICASSP , 2023. -
  20. [20][20] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov, 'Differential privacy has disparate impact on model accuracy,' Advances in NeurIPS , 2019. -
  21. [21][21] Lulu Wang, Junxiang Zheng, Yongzhi Cao, and Hanpin Wang, 'Enhance PATE on complex tasks with knowledge transferred from non-private data,' IEEE Access , pp. 50081-50094, 2019. -
  22. [22][22] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot, 'Machine unlearning,' in IEEE Symposium on Security and Privacy , 2021. -
  23. [23][23] Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangchuan Liu, 'Federated unlearning,' arXiv preprint arXiv:2012.13891 , 2020. - · arXiv
  24. [24][24] Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al., 'Measuring forgetting of memorized training examples,' arXiv preprint arXiv:2207.00099 , 2022. - · arXiv
  25. [25][25] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo, 'Knowledge unlearning for mitigating privacy risks in language models,' arXiv preprint arXiv:2210.01504 , 2022. - · arXiv
  26. [27][27] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, 'LibriSpeech: an ASR corpus based on public domain audio books,' in Proc. ICASSP , 2015. -
  27. [28][28] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher, 'Pointer sentinel mixture models,' arXiv preprint arXiv:1609.07843 , 2016. - · arXiv
  28. [29][29] Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer, 'Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition,' in Proc. ICASSP , 2021. -
  29. [30][30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al., 'Llama 2: Open foundation and fine-tuned chat models,' arXiv preprint arXiv:2307.09288 , 2023. - · arXiv
  30. [31][31] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, 'LoRA: Low-rank adaptation of large language models,' arXiv preprint arXiv:2106.09685 , 2021. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Liu, Z., & Kalinli, O.
Affiliations
Meta (United States) (US)
ORCID
Liu, Zhe
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
10261-10265
ISSN
1520-6149 (ISSN-L)
DOI
10.48550/arxiv.2309.16082
arXiv
2309.16082
OpenAlex
W4392902780
Cited by
1
References
0
Citations by year
2025: 1
FWCI
0.29
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
unlearning, language models, memorization, teacher-student models, privacy-utility trade-off, fine-tuning, LibriSpeech, WikiText-103

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review