LLM-PBE: Assessing Data Privacy in Large Language Models
Open accessThe DOI matches its Crossref record; 94 of the 94 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs) have become integral to numerous domains, significantly advancing applications in data management, mining, and analysis. Their profound capabilities in processing and interpreting complex language data, however, bring to light pressing concerns regarding data privacy, especially the risk of unintentional training data leakage. Despite the critical nature of this issue, there has been no existing literature to offer a comprehensive assessment of data privacy risks in LLMs. Addressing this gap, our paper introduces LLM-PBE, a toolkit crafted specifically for the systematic evaluation of data privacy risks in LLMs. LLM-PBE is designed to analyze privacy across the entire lifecycle of LLMs, incorporating diverse attack and defense strategies, and handling various data types and metrics. Through detailed experimentation with multiple LLMs, LLM-PBE facilitates an in-depth exploration of data privacy concerns, shedding light on influential factors such as model size, data characteristics, and evolving temporal dimensions. This study not only enriches the understanding of privacy issues in LLMs but also serves as a vital resource for future research in the field. Aimed at enhancing the breadth of knowledge in this area, the findings, resources, and our full technical report are made available at https://llm-pbe.github.io/, providing an open platform for academic and practical advancements in LLM privacy assessment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Calibrating Noise to Sensitivity in Private Data Analysis
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006) · Lecture Notes in Computer Science · doi:10.1007/11681878_14 · Cited by 7,272
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578
The Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Privacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356 · cited 1×
“DP has been used in the training of machine learning models [6, 60, 63], which is usually achieved by adding noises to gradients when using stochastic gradient descent.”
Privacy-preserving data mining
Agrawal, R., & Srikant, R. (2000) · Proceedings of the 2000 ACM SIGMOD international conference on Management of data · doi:10.1145/342009.335438 · Cited by 1,719 · cited 1×
“There have been many data privacy protection approaches [7, 11, 82, 83].”
The Enron Corpus: A New Dataset for Email Classification Research
Klimt, B., & Yang, Y. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30115-8_22 · Cited by 1,157
Data Privacy through Optimal k-Anonymization
Bayardo, R. J., & Agrawal, R. (2005) · 21st International Conference on Data Engineering (ICDE'05) · doi:10.1109/icde.2005.42 · Cited by 1,131
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 1×
“The results of using DEAs on Github are presented in Table 11, where we also evaluate CodeLlama [95], which is an LLM for code based on Llama-2.”
MethodJailbreaking Black Box Large Language Models in Twenty Queries
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2023) · arXiv · doi:10.48550/arxiv.2310.08419 · Cited by 522
Multi-step Jailbreaking Privacy Attacks on ChatGPTPreprint
Li, H., Guo, D., Fan, W., Xu, M., Huang, J., & Song, Y.-Q. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2304.05197 · Cited by 514
Membership Inference Attacks From First Principles
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., & Tramèr, F. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833649 · Cited by 502
Proceedings of the 32nd international conference on Very large data bases
Dayal, U., Whang, K.-Y., Lomet, D., Alonso, G., Lohman, G. M., Kersten, M. L., Sang, K., & Kim, Y. (2006) · Medical Entomology and Zoology · Cited by 479
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
Deep Learning Scaling is Predictable, Empirically
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017) · arXiv · doi:10.48550/arxiv.1712.00409 · Cited by 427
Are aligned neural networks adversarially aligned?Preprint
Carlini, N., Nasr, M., Choquette-Choo, C. A., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.15447 · Cited by 425
MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413
Differential Privacy via Wavelet Transforms
Xiao, X., Wang, G., & Gehrke, J. (2011) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2010.247 · Cited by 369
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291
Self-citationJailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2307.02483 · Cited by 280
Jailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3508 · Cited by 280
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Mireshghallah, F., Goyal, K., Uniyal, A., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2203.03929 · Cited by 256
Differentially private histogram publication
Xu, J., Zhang, Z., Xiao, X., Yang, Y., Yu, G., & Winslett, M. (2013) · The VLDB Journal · doi:10.1007/s00778-013-0309-y · Cited by 228
Pilán, I., Lison, P., Øvrelid, L., Papadopoulou, A. I., Sánchez, D., & Batet, M. (2022) · Computational Linguistics · doi:10.1162/coli_a_00458 · Cited by 227
Adaptive Laplace Mechanism: Differential Privacy Preservation in Deep Learning
Phan, N., Wu, X., Hu, H., & Dou, D. (2017) · 2017 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2017.48 · Cited by 217
Quantifying Memorization Across Neural Language Models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204
Universal and Transferable Adversarial Attacks on Aligned Language Models
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197
Self-citationFast Yet Effective Machine Unlearning
Tarun, A. K., Chundawat, V. S., Mandal, M., & Kankanhalli, M. (2021) · arXiv · doi:10.48550/arxiv.2111.08947 · Cited by 190
Privacy Risks of General-Purpose Language Models
Pan, X., Zhang, M., Ji, S., & Yang, M. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00095 · Cited by 188 · cited 3×
“…(e.g., data extraction attack [18, 50]) and do not cover a broad range of attack metrics; 4) Limited consideration of privacy protection approaches : Existing studies [51, 55, 59, 77, 81, 89] usually lack the consideration of the effect of using privacy protection approaches on the data leakage.”
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., Lu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.02137 · Cited by 186 · cited 2×
“In the contemporary landscape of technology, Large Language Models (LLMs) [52, 54, 66, 69] have rapidly ascended to prominence, revolutionizing the way we interact with data.”
Fast Yet Effective Machine Unlearning
Tarun, A. K., Chundawat, V. S., Mandal, M., & Kankanhalli, M. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3266233 · Cited by 182
Enhanced Membership Inference Attacks against Machine Learning Models
Ye, J., Maddi, A., Murakonda, S. K., Bindschaedler, V., & Shokri, R. (2022) · Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3548606.3560675 · Cited by 181
Explaining neural scaling laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311878121 · Cited by 181
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01373 · Cited by 160 · cited 1×
“To investigate the effect of data type on privacy risks, we use DEAs with ECHR dataset on Llama-2 7b [69], which includes different types of PII types including name, location, and date.”
On the Importance of Difficulty Calibration in Membership Inference AttacksPreprint
Watson, L., Guo, C., Cormode, G., & Sablayrolles, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08440 · Cited by 134
Natural language to SQL: where are we today?
Kim, H., So, B.-H., Han, W.-S., & Lee, H. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3401960.3401970 · Cited by 123
KGA: A General Machine Unlearning Framework Based on Knowledge Gap AlignmentPreprint
Wang, L., Chen, T., Yuan, W., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.06535 · Cited by 121
Machine Unlearning of Features and Labels
Warnecke, A., Pirch, L., Wressnegger, C., & Rieck, K. (2021) · arXiv · doi:10.48550/arxiv.2108.11577 · Cited by 119
Considerations for Differentially Private Learning with Large-Scale Public PretrainingPreprint
Tramèr, F., Kamath, G., & Carlini, N. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.06470 · Cited by 119
How Large Language Models Will Disrupt Data Management
Fernandez, R. C., Elmore, A. J., Franklin, M. J., Krishnan, S., & Tan, C. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3611479.3611527 · Cited by 115
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2302.05733 · Cited by 114 · cited 2×
“2) Splitting-based methods: attackers split the attack keywords into multiple sub-parts so that LLMs cannot detect them [59].”
MethodIgnore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110 · cited 9×
“PLAs [35, 59] aim to steal system or user prompts from LLMs.”
InfluentialDecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11698 · Cited by 110
Self-citationMembership Inference Attack Susceptibility of Clinical Language ModelsPreprint
Jagannatha, A., Rawat, B. P. S., & Yu, H. Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08305 · Cited by 108
DecodingTrust: A comprehensive assessment of trustworthiness in GPT models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1361 · Cited by 107
Self-citationBag of Tricks for Training Data Extraction from Language ModelsPreprint
Yu, W.-C., Pang, T., Liu, Q., Du, C., Kang, B.-Y., Huang, Y., Lin, M., & Yan, S.-C. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2302.04460 · Cited by 104
Differentially Private Fine-tuning of Language Models
Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.06500 · Cited by 101
Scalable Extraction of Training Data from (Production) Language Models
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., & Lee, K. (2023) · arXiv · doi:10.48550/arxiv.2311.17035 · Cited by 96
ERM-KTP: Knowledge-Level Machine Unlearning via Knowledge Transfer
Lin, S., Zhang, X., Chen, C., Chen, X., & Susilo, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01929 · Cited by 86
Differentially Private Decoding in Large Language ModelsPreprint
Majmudar, J., Dupuy, C., Peris, C., Smaili, S., Gupta, R., & Zemel, R. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.13621 · Cited by 80
Zhang, X., Malkov, Y., Florez, O., Park, S., McWilliams, B., Han, J., & El-Kishky, A. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599921 · Cited by 76
Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation
Tang, X., Shin, R., Inan, H. A., Manoel, A., Mireshghallah, F., Lin, Z., Gopi, S., Kulkarni, J., & Sim, R. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.11765 · Cited by 75
Formation of SQL from Natural Language Query using NLP
Uma, M., Sneha, V., Sneha, G., Bhuvana, J., & Bharathi, B. (2019) · International Conference on Computational Intelligence in Data Science (ICCIDS) · doi:10.1109/iccids.2019.8862080 · Cited by 64
Privacy-Preserving In-Context Learning for Large Language ModelsPreprint
Wu, T., Panda, A., Wang, J. T., & Mittal, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.01639 · Cited by 60
CatSQL : Towards Real World Natural Language to SQL Applications
Fu, H., Liu, C., Wu, B., Li, F., Tan, J., & Sun, J. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3583140.3583165 · Cited by 57
Self-citation(2023) · doi:10.52202/075280-2687 · Cited by 57
Counterfactual Memorization in Neural Language Models
Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramèr, F., & Carlini, N. (2021) · arXiv · doi:10.48550/arxiv.2112.12938 · Cited by 54
maxbachmann/RapidFuzz: Release 1.5.0
Bachmann, M., layday, thomasryde, dotlambda, Kokkinou, G., Schreiner, H., Fihl-Pearson, J., dheeraj, Vioshim, pekkarr, et al. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5228985 · Cited by 52
An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models
Mireshghallah, F., Uniyal, A., Wang, T., Evans, D., & Berg-Kirkpatrick, T. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.119 · Cited by 47
Shao, H., Huang, J., Zheng, S., & Chang, K. (2023) · Findings · doi:10.48550/arxiv.2305.12707 · Cited by 44
Detecting Pretraining Data from Large Language Models
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2310.16789 · Cited by 37
From Large Language Models to Databases and Back: A Discussion on Research and Education
Amer-Yahia, S., Bonifati, A., Chen, L., Li, G., Shim, K., Xu, J., & Yang, X. (2023) · ACM SIGMOD Record · doi:10.1145/3631504.3631518 · Cited by 34
Chalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Αλέτρας, Ν., Androutsopoulos, I., & Malakasiotis, P. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.48550/arxiv.2103.13084 · Cited by 34
CAM: A Large Language Model-based Creative Analogy Mining Framework
Bhavya, B., Xiong, J., & Zhai, C. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3587431 · Cited by 32
Selective Pre-training for Private Fine-tuning
Yu, D., Gopi, S., Kulkarni, J., Lin, Z., Naik, S., Religa, T. L., Yin, J., & Zhang, H. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2305.13865 · Cited by 27
Do Membership Inference Attacks Work on Large Language Models?
Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., & Hajishirzi, H. (2024) · arXiv · doi:10.48550/arxiv.2402.07841 · Cited by 17
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
Yuan, Y., Jiao, W., Wang, W., Huang, J.-T., He, P., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.06463 · Cited by 14
Mireshghallah, N., Kim, H., Zhou, X., Tsvetkov, Y., Sap, M., Shokri, R., & Choi, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.17884 · Cited by 14 · cited 3×
“…(e.g., data extraction attack [26, 81]) and do not cover a broad range of attack metrics; 4) Limited consideration of privacy protection approaches : Existing studies [82, 88, 92, 114, 118, 133] usually lack the consideration of the effect of using privacy protection approaches on the data leakage.”
BackgroundOmniscientDB: A Large Language Model-Augmented DBMS That Knows What Other DBMSs Do Not Know
Urban, M., Nguyen, D., & Binnig, C. (2023) · International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (aiDM) · doi:10.1145/3593078.3593933 · Cited by 12
Memorization in NLP Fine-tuning MethodsPreprint
Mireshghallah, F., Uniyal, A., Wang, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12506 · Cited by 11
Fu, W., Wang, H., Gao, C., Liu, G., Li, Y., & Jiang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.06062 · Cited by 9
Zero redundancy distributed learning with differential privacyPreprint
Bu, Z., Chiu, J., Liu, R., Zha, S., & Karypis, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.11822 · Cited by 9
Effective Prompt Extraction from Language Models
Zhang, Y., Carlini, N., & Ippolito, D. (2023) · arXiv · doi:10.48550/arxiv.2307.06865 · Cited by 8
On the accuracy and efficiency of group-wise clipping in differentially private optimizationPreprint
Bu, Z., Liu, R., Wang, Y., Zha, S., & Karypis, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19215 · Cited by 8
ER-AE: Differentially Private Text Generation for Authorship AnonymizationPreprint
Bo, H., Ding, S. H. H., Fung, B. C. M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08736 · Cited by 7
Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language ModelsPreprint
Duan, H., Dziedzic, A., Papernot, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15594 · Cited by 6
Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data
Treutlein, J., Choi, D., Betley, J., Marks, S., Anil, C., Grosse, R., & Evans, O. (2024) · arXiv · doi:10.48550/arxiv.2406.14546 · Cited by 6
maxbachmann/RapidFuzz: Release 1.8.0
Bachmann, M. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5584996 · Cited by 6
Membership Inference Attacks against Language Models via Neighbourhood Comparison
Mattern, J., Mireshghallah, F., Jin, Z., Schölkopf, B., Sachan, M., & Berg-Kirkpatrick, T. (2023) · arXiv · doi:10.48550/arxiv.2305.18462 · Cited by 4
Knowledge Unlearning for Mitigating Privacy Risks in Language Models
Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., & Seo, M. (2022) · arXiv · doi:10.48550/arxiv.2210.01504 · Cited by 4
Combing for Credentials: Active Pattern Extraction from Smart ReplyPreprint
Jayaraman, B., Ghosh, E., Chase, M., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.10802 · Cited by 3
Differential Privacy for Text Analytics via Natural Text SanitizationPreprint
Yue, X., Du, M., Wang, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01221 · Cited by 2
PLeak: Prompt Leaking Attacks against Large Language Model ApplicationsPreprint
Hui, B., Yuan, H., Gong, N. Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.06823 · Cited by 2
A Synthetic Dataset for Personal Attribute InferencePreprint
Yukhymenko, H., Staab, R., Vero, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.07217
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Research works printed but not linked: 41
- [18] Angela Bonifati, Sihem Amer-Yahia, Chen Lei, Li Guoliang, Shim Kyuseok, Xu Jianliang, and Yang Xiaochun. 2023. From Large Language Models to Databases and Back A discussion on research and education. SIGMOD record (2023). -
- [22] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. 2023. Automatic clipping: Differentially private deep learning made easier and stronger. Conference on Neural Information Processing Systems (2023). -
- [23] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. 2023. Differentially private bias-term only fine-tuning of foundation models. In The Eleventh International Conference on Learning Representations . -
- [24] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. 2023. Differentially private optimization on large model at small cost. In International Conference on Machine Learning . PMLR, 3192-3218. -
- [27] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying Memorization Across Neural Language Models. In The Eleventh International Conference on Learning Representations . https://openreview.net/forum?id=TatRHT\_1cK Accessed: 2024-07-16. -
- [28] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In 28th USENIX Security Symposium, USENIX Security 2019 . -
- [30] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel HerbertVoss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) . 2633-2650. -
- [31] Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in English. arXiv preprint arXiv:1906.02059 (2019). - · arXiv
- [35] Alfredo Cuzzocrea and Hossain Shahriar. 2017. Data masking techniques for NoSQL database security: A systematic review. In 2017 IEEE International Conference on Big Data (Big Data) . IEEE, 4467-4473. -
- [39] Cynthia Dwork. 2006. Differential privacy. In International colloquium on automata, languages, and programming . Springer, 1-12. -
- [42] Vitaly Feldman and Chiyuan Zhang. 2020. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems 33 (2020), 2881-2891. -
- [47] Tanishq Gupta, Mohd Zaki, NM Anoop Krishnan, and Mausam. 2022. MatSciBERT: A materials domain language model for text mining and information extraction. npj Computational Materials 8, 1 (2022), 102. -
- [51] Sorami Hisamoto, Matt Post, and Kevin Duh. 2020. Membership inference attacks on sequence-to-sequence models: Is my data in your machine translation system? Transactions of the Association for Computational Linguistics 8 (2020), 49-63. -
- [52] Junyuan Hong, Jiachen T Wang, Chenhui Zhang, LI Zhangheng, Bo Li, and Zhangyang Wang. 2024. DP-OPT: Make Large Language Model Your PrivacyPreserving Prompt Engineer. In The Twelfth International Conference on Learning Representations . -
- [54] Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are Large PreTrained Language Models Leaking Your Personal Information? EMNLP Findings (2022). -
- [60] Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning . PMLR, 10697-10707. -
- [65] Yannis Kotidis and Nick Roussopoulos. 1999. Dynamat: A dynamic view management system for data warehouses. ACMSigmod record 28, 2 (1999), 371-382. -
- [66] Yannis Kotidis and Nick Roussopoulos. 2001. A case for dynamic view management. ACM Transactions on Database Systems (TODS) 26, 4 (2001), 388-423. -
- [67] Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. 2023. Privacy Adhering Machine Un-learning in NLP. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings) . 268-277. -
- [68] Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C Wallace. 2021. Does BERT pretrained on clinical notes reveal sensitive data? arXiv preprint arXiv:2104.07762 (2021). - · arXiv
- [70] Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin-Tat Lee, and Abhradeep Guha Thakurta. 2022. When Does Differentially Private Learning Not Suffer in High Dimensions? Advances in Neural Information Processing Systems 35 (2022), 28616-28630. -
- [71] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2022. Large language models can be strong differentially private learners. The Tenth International Conference on Learning Representations (2022). -
- [72] Yansong Li, Zhixing Tan, and Yang Liu. 2023. Privacy-preserving prompt tuning for large language model services. arXiv preprint arXiv:2305.06212 (2023). - · arXiv
- [75] Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing leakage of personally identifiable information in language models. IEEE Symposium on Security and Privacy (S&P) (2023). -
- [77] Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. 2022. Differentially Private Language Models for Secure Data Sharing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 4860-4873. https://aclanthology.org/2022.emnlpmain.323 Accessed: 2024-07-16. -
- [79] Natalie Maus, Patrick Chao, Eric Wong, and Jacob R Gardner. 2023. Black box adversarial prompting for foundation models. In The Second Workshop on New Frontiers in Adversarial Machine Learning . -
- [80] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. 2019. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP) . IEEE, 691-706. -
- [86] Sharan Narang and Aakanksha Chowdhery. 2022. Pathways language model (palm): Scaling to 540 billion parameters for breakthrough performance. Google AI Blog (2022). -
- [88] Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. 2021. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on security and privacy (SP) . IEEE, 866-882. -
- [92] Ashwinee Panda, Christopher A. Choquette-Choo, Zhengming Zhang, Yaoqing Yang, and Prateek Mittal. 2024. Teach LLMs to Phish: Stealing Private Information from Language Models. In The Twelfth International Conference on Learning Representations . https://openreview.net/forum?id=qo21ZlfNu6 Accessed: 2024-07-16. -
- Ashwinee Panda, Zhengming Zhang, Yaoqing Yang, and Prateek Mittal. 2023. Teach GPT To Phish. In The Second Workshop on New Frontiers in Adversarial Machine Learning. https://openreview.net/forum?id=tGvWCD9BEP Accessed: 2024-07-16.
- [98] Ricardo Jorge Santos, Jorge Bernardino, and Marco Vieira. 2011. A data masking technique for data warehouses. In Proceedings of the 15th Symposium on International Database Engineering & Applications . 61-69. -
- [103] Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. 2024. Beyond Memorization: Violating Privacy via Inference with Large Language Models. In The Twelfth International Conference on Learning Representations . https: //openreview.net/forum?id=kmn0BhQk7p Accessed: 2024-07-16. -
- [110] Immanuel Trummer. 2023. From bert to gpt-3 codex: harnessing the potential of very large language models for data management. arXiv preprint arXiv:2306.09339 (2023). - · arXiv
- [113] Thomas Vakili and Hercules Dalianis. 2021. Are Clinical BERT Models Privacy Preserving? The Difficulty of Extracting Patient-Condition Associations.. In HUMAN@ AAAI Fall Symposium . -
- [114] Yuli Vasiliev. 2020. Natural language processing with Python and spaCy: A practical introduction . No Starch Press. -
- [115] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). -
- [130] Xiang Yue, Huseyin A Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Huan Sun, David Levitan, and Robert Sim. 2023. Synthetic text generation with differential privacy: A simple and practical recipe. ACL (2023). -
- [136] Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. 2022. Prompt certified machine unlearning with randomized gradient smoothing and quantization. Advances in Neural Information Processing Systems 35 (2022), 1343313455. -
- [137] Xuandong Zhao, Lei Li, and Yu-Xiang Wang. 2022. Provably Confidential Language Modelling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . 943-955. -
- [138] Xuanhe Zhou, Zhaoyan Sun, and Guoliang Li. 2024. DB-GPT: Large Language Model Meets Database. Data Science and Engineering (2024), 1-10. -
Cited by
Privacy-Preserving Techniques in Generative AI and Large Language Models: A Narrative Review
Feretzakis, G., Papaspyridis, K., Gkoulalas-Divanis, A., & Verykios, V. S. (2024) · Information · doi:10.3390/info15110697 · Cited by 145
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and DeploymentPreprint
Wang, K., Zhang, G., Zhou, Z., Wu, J., Yu, M., Zhao, S., Yin, C., Fu, J., Yan, Y., Luo, H., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.15585 · Cited by 196
Self-citationUnderstanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 169
A Survey of Small Language ModelsPreprint
Nguyen, C. V., Shen, X., Aponte, R., Xia, Y., Basu, S., Hu, Z., Chen, J., Parmar, M., Kunapuli, S., Barrow, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.20011 · Cited by 120
A survey on privacy risks and protection in large language models
Chen, K., Zhou, X., Lin, Y., Feng, S., Shen, L., & Wu, P. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00177-1 · Cited by 101
Metadata
- Authors
- Li, Q., Hong, J., Xie, C., Tan, J., Xin, R., Hou, J., Yin, X., Wang, Z., Hendrycks, D., Wang, Z., Li, B., He, B., & Song, D.
- Affiliations
- University of California, Berkeley (US)
- The University of Texas at Austin (US)
- University of Illinois Urbana-Champaign (US)
- National University of Singapore (SG)
- Center for AI Safety
- University of Chicago (US)
- ORCID
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2024-07
- Volume
- 17
- Issue
- 11
- Pages
- 3201-3214
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W4402042869
- Cited by
- 42
- References
- 32
- Citations by year
- 2026: 20 · 2025: 21 · 2024: 1
- FWCI
- 6.21
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the VLDB Endowment (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2229876, 2212176)
- U.S. Department of Homeland Security
- National Research Foundation
- National Research Foundation Singapore
- Info-communications Media Development Authority
- Defense Advanced Research Projects Agency (2229876)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Keywords
- large language models, data privacy, training data leakage, privacy risk assessment, model size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.