CoChat
Sign Up
Cited by 165Privacy-Preserving Technologies in DataEthics and Social Impacts of AIPrivacy, Security, and Data Protection

What Does it Mean for a Language Model to Preserve Privacy?

Open access
Brown, Hannah, Lee, Katherine, Mireshghallah, Fatemehsadat, Shokri, Reza, Tramèr, Florian·2022·2022 ACM Conference on Fairness Accountability and Transparency
DOI 10.1145/3531146.3534642
Open this paper in CoChat — free accountOpen PDF

Abstract

Natural language reflects our private lives and identities, making its privacy concerns as broad as those of real life. Language models lack the ability to understand the context and sensitivity of text, and tend to memorize phrases present in their training sets. An adversary can exploit this tendency to extract training data. Depending on the nature of the content and the context in which this data was collected, this could violate expectations of privacy. Thus, there is a growing interest in techniques for training language models that preserve privacy. In this paper, we discuss the mismatch between the narrow assumptions made by popular data protection techniques (data sanitization and differential privacy), and the broadness of natural language and of privacy as a social norm. We argue that existing protection methods cannot guarantee a generic and meaningful notion of privacy for language models. We conclude that language models should be trained on text data which was explicitly produced for public use.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3531146.3534642

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Using learner-generated data without explicit consent or adequate anonymisation raises serious issues about exposing sensitive information.

    “Using learner-generated data without explicit consent or adequate anonymisation raises serious issues about exposing sensitive information [82].”

    From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [82]

    The Introduction and §6 explain that using personal text to train language models can violate privacy and that informed consent is difficult to obtain. Section 3.0.2 describes sensitive information being extracted even from a model trained on de-identified clinical notes. The paper does not address learner-generated data specifically, so the claim extends its warning to an educational setting. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • This lack of attention to privacy issues is particularly concerning as LLMs-based innovations work with stakeholders' natural languages that may contain personal and sensitive information regarding their private lives and identities.

    “This lack of attention to privacy issues is particularly concerning as LLMs-based innovations work with stakeholders' natural languages that may contain personal and sensitive information regarding their private lives and identities (Brown et al., 2022).”

    From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Brown et al., 2022

  • This usage could potentially expose private information as LLMs are incapable of understanding the context and sensitivity of text, and thus could return stakeholders' personal information based on semantic relationships.

    “This usage could potentially expose private information as LLMs are incapable of understanding the context and sensitivity of text, and thus, could return stakeholders' personal information based on semantic relationships (Brown et al., 2022).”

    From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Brown et al., 2022

  • Deidentifying data for LLMs is challenging.

    “For example, deidentifying data for LLMs is challenging [60].”

    From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [60]

  • The viral nature of deepfakes renders removal efforts challenging and potentially ineffective.

    “Additionally, the viral nature of deepfakes, easily disseminated and replicated across platforms and jurisdictions, renders removal e ff orts challenging and potentially ine ff ective (Brown et al. 2022).”

    From Generative AI and deepfakes: a human rights approach to tackling harmful content (2024) · cites this paper as Brown et al. 2022

    Our copy discusses the difficulty of removing sensitive information from text datasets (Section 5.1.1) and how language models can retain earlier versions of deleted online text (Section 6). Neither passage addresses deepfakes, their spread across platforms or jurisdictions, or the effectiveness of removing them. Searches for deepfakes and removal of rapidly shared synthetic media found no passage supporting the attributed claim. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 79

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. ACM, Vienna, Austria, 308-318.
  2. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differentially private BERT. arXiv preprint arXiv:2108.01624 abs/2108.01624 (2021). · arXiv
  3. Tuomas Aura, Thomas A Kuhn, and Michael Roe. 2006. Scanning electronic documents for personally identifiable information. In Proceedings of the 5th ACM workshop on Privacy in electronic society. ACM, New York, NY, United States, 41-50.
  4. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. ACL, Lisbon, Portugal, 632-642.
  5. Penelope Brown, Stephen C Levinson, and Stephen C Levinson. 1987. Polite- ness: Some universals in language usage. Vol. 4. Cambridge university press, Cambridge, United Kingdom.
  6. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 abs/2005.14165 (2020). · arXiv
  7. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th {USENIX} Security Symposium ({USENIX} Security 19). USENIX Association, Santa Clara, CA, 267-284.
  8. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2020. Extracting training data from large language models. arXiv preprint arXiv:2012.07805 abs/2012.07805 (2020). · arXiv
  9. Amanda Cercas Curry and Verena Rieser. 2018. #MeToo Alexa: How Con- versational Systems Respond to Sexual Harassment. In Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing. Association for Computational Linguistics, New Orleans, Louisiana, USA, 7-14. https: //doi.org/10.18653/v1/W18-0802 · DOI
  10. Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. 2011. Differen- tially private empirical risk minimization. Journal of Machine Learning Research 12, 3 (2011), 1069-1109.
  11. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 abs/2107.03374 (2021). · arXiv
  12. Rishav Chourasia, Jiayuan Ye, and Reza Shokri. 2021. Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descent. In Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, Online.
  13. Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. 2017. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association 24, 3 (2017), 596-606.
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 abs/1810.04805 (2018). · arXiv
  15. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. 2021. Documenting the English Colossal Clean Crawled Corpus. ArXiv abs/2104.08758 (2021). · arXiv
  16. Paul Dourish. 2004. What We Talk About When WeTalk About Context. Personal and Ubiquitous Computing 8 (2004), 19-39.
  17. Greg Durrett and Dan Klein. 2015. Neural CRF Parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, 302-312. https://doi.org/10.3115/v1/P15-1030 · DOI
  18. Cynthia Dwork. 2008. Differential privacy: A survey of results. In International conference on theory and applications of models of computation. Springer, Xi'an, China, 1-19.
  19. Cynthia Dwork. 2011. The promise of differential privacy a tutorial on algo- rithmic techniques. In 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, D (Oct. 2011). IEEE, Washington, DC, United States, 1-2.
  20. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Cali- brating noise to sensitivity in private data analysis. In Theory of cryptography conference. Springer, New York, United States, 265-284.
  21. Jacob Eisenstein, Brendan O'Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change in social media. PloS one 9, 11 (2014), e113114.
  22. Vitaly Feldman. 2020. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing. ACM, Online, 954-959.
  23. Simson Garfinkel, John M Abowd, and Christian Martindale. 2019. Understand- ing database reconstruction attacks on public data. Commun. ACM 62, 3 (2019), 46-53.
  24. Margaret Goss. 2020. Temporal News Frames and Judgment: The Hillary Clinton Email Scandal. Ph.D. Dissertation. Carnegie Mellon University.
  25. Andrea Greve, Elisa Cooper, Roni Tibon, and Richard N. Henson. 2019. Knowl- edge is power: Prior knowledge aids memory for both congruent and incongru- ent events, but in different ways. Journal of Experimental Psychology: General 148, 2 (Feb. 2019), 325-341. https://doi.org/10.1037/xge0000498 · DOI
  26. Bridget Haire, Christy E. Newman, and Bianca Fileborn. 2019. Shitty Media Men. In #MeToo and the Politics of Social Change, Bianca Fileborn and Rachel Loney-Howes (Eds.). Springer International Publishing, Cham, 201-216. https: //doi.org/10.1007/978-3-030-15213-0_13 · DOI
  27. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-Trained Models: Past, Present and Future. AI Open 10.1016/j.aiopen.2021.08.002 (2021). https://doi.org/10. 1016/j.aiopen.2021.08.002 · DOI · Paper page
  28. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. ACM, New Orleans, LA, 123-129.
  29. Chaya Hiruncharoenvate, Zhiyuan Lin, and Eric Gilbert. 2015. Algorithmically Bypassing Censorship on Sina Weibo with Nondeterministic Homophone Substi- tutions. Proceedings of the International AAAI Conference on Web and Social Media 9, 1 (2015), 150-158. https://ojs.aaai.org/index.php/ICWSM/article/view/14637 Number: 1.
  30. Shlomo Hoory, Amir Feder, Avichai Tendler, Alon Cohen, Sofia Erell, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, et al. 2021. Learning and Evaluating a Differentially Private Pre-trained Language Model. In Proceedings of the Third Workshop on Privacy in Natural Language Processing. ACL, Online, 21-29.
  31. Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In The 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 588-602.
  32. Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: Towards machine ethics and norms. arXiv preprint arXiv:2110.07574 abs/2104.08758 (2021). · arXiv
  33. Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific data 3, 1 (2016), 1-9.
  34. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations. ICLR, Online.
  35. Soomin Kim, Changhoon Oh, Won Ik Cho, Donghoon Shin, Bongwon Suh, and Joonhwan Lee. 2021. Trkic G00gle: Why and How Users Game Translation Algorithms. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (Oct. 2021), 344:1-344:24. https://doi.org/10.1145/3476085 · DOI
  36. Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In European Conference on Machine Learning. Springer, Pisa, Italy, 217-226.
  37. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499 abs/2107.06499 (2021). · arXiv
  38. Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wal- lace. 2021. Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? arXiv:2104.07762 [cs.CL] · arXiv
  39. Daniel Levy, Ziteng Sun, Kareem Amin, Satyen Kale, Alex Kulesza, Mehryar Mohri, and Ananda Theertha Suresh. 2021. Learning with User-Level Privacy. arXiv preprint arXiv:2102.11845 abs/2102.11845 (2021). · arXiv
  40. Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A Persona-Based Neural Conversation Model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 994-1003. https://doi.org/10.18653/v1/P16-1094 · DOI · Paper page
  41. Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679 abs/2110.05679 (2021). · arXiv
  42. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 abs/1907.11692 (2019). · arXiv
  43. Yunhui Long, Vincent Bindschaedler, and Carl A. Gunter. 2017. Towards Mea- suring Membership Privacy. ArXiv abs/1712.09136 (2017). · arXiv
  44. Junyu Lu, Xiancong Ren, Yazhou Ren, Ao Liu, and Zenglin Xu. 2020. Im- proving Contextual Language Models for Response Retrieval in Multi-Turn Conversation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR '20). Association for Computing Machinery, New York, NY, USA, 1805-1808. https://doi.org/10.1145/3397271.3401255 · DOI
  45. Saeed Mahloujifar, Huseyin A Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa. 2021. Membership Inference on Word Embedding and Beyond. arXiv preprint arXiv:2106.11384 abs/2106.11384 (2021). · arXiv
  46. Bodhisattwa Prasad Majumder, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Ju- lian McAuley. 2020. Like hiking? You probably enjoy nature: Persona-grounded Dialog with Commonsense Expansions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 9194-9206. https://doi.org/10.18653/v1/ 2020.emnlp-main.739 · DOI
  47. Alice E Marwick and danah boyd. 2014. Networked privacy: How teenagers negotiate context in social media. New Media & Society 16, 7 (Nov. 2014), 1051- 1067. https://doi.org/10.1177/1461444814543995 Publisher: SAGE Publications. · DOI · Paper page
  48. H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2017. Learn- ing differentially private language models without losing accuracy. arXiv preprint arXiv:1710.06963 abs/1710.06963 (2017). · arXiv
  49. Jamshed Memon, Maira Sami, Rizwan Ahmed Khan, and Mueen Uddin. 2020. Handwritten optical character recognition (OCR): A comprehensive systematic literature review (SLR). IEEE Access 8 (2020), 142642-142668.
  50. Adam S. Miner, Liliana Laranjo, and A. Baki Kocaballi. 2020. Chatbots in the fight against the COVID-19 pandemic. npj Digital Medicine 3, 1 (May 2020), 1-4. https://doi.org/10.1038/s41746-020-0280-0 Bandiera_abtest: a Cc_license_type: cc_by Cg_type: Nature Research Journals Number: 1 Primary_atype: Com- ments & Opinion Publisher: Nature Publishing Group Subject_term: Epidemiol- ogy;Population screening Subject_term_id: epidemiology;population-screening. · DOI · Paper page
  51. Sasi Kumar Murakonda and Reza Shokri. 2020. ML Privacy Meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. Workshop on Hot Topics in Privacy Enhancing Technologies (HotPETs) 1 (2020).
  52. Arvind Narayanan and Vitaly Shmatikov. 2008. Robust de-anonymization of large sparse datasets. In 2008 IEEE Symposium on Security and Privacy (sp 2008). IEEE, Oakland, CA, USA, 111-125.
  53. Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP). IEEE, San Francisco, CA, United States, 739-753.
  54. Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin. 2021. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP). IEEE, San Francisco, 866-882.
  55. Helen Nissenbaum. 2009. Privacy in context. Stanford University Press, Stanford, CA.
  56. Alicia L. Nobles, Eric C. Leas, Theodore L. Caputi, Shu-Hong Zhu, Steffanie A. Strathdee, and John W. Ayers. 2020. Responses to addiction help-seeking from Alexa, Siri, Google Assistant, Cortana, and Bixby intelligent virtual assistants. npj Digital Medicine 3, 1 (Jan. 2020), 1-3. https://doi.org/10.1038/s41746-019- 0215-9 Bandiera_abtest: a Cc_license_type: cc_by Cg_type: Nature Research Journals Number: 1 Primary_atype: Research Publisher: Nature Publishing Group Subject_term: Epidemiology;Rehabilitation Subject_term_id: epidemiol- ogy;rehabilitation. · DOI · Paper page
  57. Beau Norgeot, Kathleen Muenzen, Thomas A Peterson, Xuancheng Fan, Ben- jamin S Glicksberg, Gundolf Schenk, Eugenia Rutenberg, Boris Oskotsky, Marina Sirota, Jinoos Yazdany, et al. 2020. Protected Health Information filter (Philter): accurately and securely de-identifying free-text clinical notes. NPJ digital medicine 3, 1 (2020), 1-8.
  58. Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 abs/1904.08779 (2019). · arXiv
  59. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  60. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Pow- ell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gri- bovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Mas- son d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv:2112.11446 [cs.CL] · arXiv
  61. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 abs/1910.10683 (2019). · arXiv
  62. Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. 2020. Training production language models without memorizing user data. arXiv preprint arXiv:2009.10031 abs/2009.10031 (2020). · arXiv
  63. Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. 2020. Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning. In 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, Online, 1291-1308. https://www.usenix.org/conference/ usenixsecurity20/presentation/salem
  64. Ahmed Salem, Yang Zhang, Mathias Humbert, Mario Fritz, and Michael Backes. 2018. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. ArXiv abs/1806.01246 (2018). · arXiv
  65. Dave Sayers. 2014. The mediated innovation model: A framework for researching media influence in language change. Journal of So- ciolinguistics 18, 2 (2014), 185-212. https://doi.org/10.1111/josl.12069 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/josl.12069 · DOI
  66. Virat Shejwalkar, Huseyin A Inan, Amir Houmansadr, and Robert Sim. 2021. Membership Inference Attacks Against NLP Classification Models. In NeurIPS 2021 Workshop Privacy in Machine Learning. NeurIPS, Online.
  67. Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. 2021. Selective Differential Privacy for Language Modeling. arXiv preprint arXiv:2108.12944 abs/2108.12944 (2021). · arXiv
  68. Congzheng Song and Ananth Raghunathan. 2020. Information leakage in em- bedding models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. ACM, Online, 377-390.
  69. Congzheng Song and Vitaly Shmatikov. 2018. The Natural Auditor: How To Tell If Someone Used Your Words To Train Their Model. ArXiv abs/1811.00513 (2018). · arXiv
  70. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and cognition. Vol. 142. Wiley-Blackwell, Hoboken, New Jersey, United States.
  71. Yingnian Tao. 2021. Who should apologise: Expressing criticism of public figures on Chinese social media in times of COVID-19. Discourse & Society 32, 5 (Sept. 2021), 622-638. https://doi.org/10.1177/09579265211013116 Publisher: SAGE Publications Ltd. · DOI
  72. Julien Tourille, Matthieu Doutreligne, Olivier Ferret, Aurélie Névéol, Nicolas Paris, and Xavier Tannier. 2018. Evaluation of a sequence tagging tool for biomedical texts. In proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis. ACL, Brussels, Belgium, 193-203.
  73. Sabine Trepte. 2021. The Social Media Privacy Model: Privacy and Communi- cation in the Light of Social Media Affordances. Communication Theory 31, 4 (Nov. 2021), 549-570. https://doi.org/10.1093/ct/qtz035 · DOI
  74. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. NeurIPS, Long Beach, CA, 5998-6008.
  75. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from Language Models. arXiv:2112.04359 [cs.CL] · arXiv
  76. Ashima Yadav and Dinesh Kumar Vishwakarma. 2020. Sentiment analysis using deep learning architectures: a review. Artificial Intelligence Review 53, 6 (2020), 4335-4385.
  77. Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. 2021. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500 abs/2110.06500 (2021). · arXiv
  78. Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. 2020. Analyzing Information Leakage of Updates to Natural Language Models. Association for Computing Machinery, New York, NY, USA, 363-375. https://doi.org/10.1145/ 3372297.3417880 · DOI · Paper page
  79. Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. 2021. Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021. Association for Computational Linguistics, Online, 3813-3827. https://doi.org/10.18653/v1/2021.findings-acl.334 · DOI

Cited by

Sorted by citations

Authors
Brown, H., Lee, K., Mireshghallah, F., Shokri, R., & Tramèr, F.
Affiliations
  • National University of Singapore (SG)
  • Cornell University (US)
  • University of California San Diego (US)
  • Google (Switzerland) (CH)
ORCID
Venue
2022 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2022-06-20
Pages
2280-2292
DOI
10.1145/3531146.3534642
OpenAlex
W4283172211
Cited by
165
References
9
Citations by year
2026: 20 · 2025: 55 · 2024: 54 · 2023: 29 · 2022: 7
FWCI
20.05
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • NUS Early Career Research Award (NUS ECRA) (NUS ECRA FY19 P16)
  • NUS Presidential Young Professorship research fund
  • VMWare Early Career Faculty Grant
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
Keywords
language models, privacy preservation, data sanitization, differential privacy, training data extraction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review