The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Analyzing Information Leakage of Updates to Natural Language Models
Open accessAbstract
To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update can reveal a surprising amount of detailed information about changes in the training data. We propose two new metrics---differential score and differential rank---for analyzing the leakage due to updates of natural language models. We perform leakage analysis using these metrics across models trained on several different datasets using different methods and configurations. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recently, existing research has shown that the unlearning operation not only does not reduce the risk of user privacy leakage but actually increases this risk [106, 107].”
From Machine Unlearning: A Survey (2024) · cites this paper as [106]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Training data extraction attacks have until now been limited to small LMs trained on academic datasets under artificial training setups (e.g., for more epochs than typical) [8, 66, 68, 73], or settings where the adversary has a priori knowledge of the secret they want to extract (e.g., a social security number) [8, 27].”
From Extracting Training Data from Large Language Models (2020) · cites this paper as [73]
Experimental — accuracy not yet measuredIn the citing paper
“Prior work shows that (100 × to 1000 × smaller) language models potentially memorize training data in semi-realistic settings [8, 73].”
From Extracting Training Data from Large Language Models (2020) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“The most prominent are Recurrent Neural Networks (RNNs) using Long Short-Term Memory [10] cells (or variants thereof) and the more recent Transformers [15, 21].”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“The most prominent are Recurrent Neural Networks (RNNs) using Long Short-Term Memory [17] cells (or variants thereof) and the more recent Transformers [23, 30].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“(2) a model based on the Transformer architecture [21] (more concretely, using the BERT [6] codebase) with four layers of six attention heads, each with a hidden dimension of 192.”
BackgroundMethodBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · cited 1×
“At a high level, differential privacy can be achieved in gradientbased optimization computations [1, 3, 20] by clipping the gradient of every record in a batch according to some bound L, then adding noise proportional to L to the sum of the clipped gradients, averaging over the batch size and using this noisy average…”
BackgroundThe Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557 · cited 2×
“Recall that an ε-DP algorithm F is kε-differentially private for groups of size k [8].”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 2×
“data (unlike [18]) and the auditor only observes a list of several top-ranked tokens.”
BackgroundThe Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2013) · doi:10.1561/9781601988195 · Cited by 3,856
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961 · cited 1×
“…reconstruction attacks [24] against the updated model, as the The goal of membership inference attacks [25, 26] is weaker in Finally, model inversion attacks [12, 13] repurpose a model to work backwards , inferring unknown attributes of individuals given Relationship to differential privacy.…”
BackgroundAdvances in Neural Information Processing Systems 32 (NeurIPS 2019)
Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché–Buc, F., Fox, E. B., & Garnett, R. (2020) · HAL · Cited by 2,451
Research works printed but not linked: 4
- Bourtoule Lucas, 2021, Machine Unlearning. In 42nd IEEE Symposium on Security and Privacy, S&P
- Dwork Cynthia, 2010, ICS
- Guo Chuan, 2020, 37th International Conference on Machine Learning, ICML
- Vaswani Ashish, 2017, NIPS
Cited by
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
“Prior work shows that (100 × to 1000 × smaller) language models potentially memorize training data in semi-realistic settings [8,73].”
BackgroundSecurity and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445
Machine Unlearning of Features and Labels
Warnecke, A., Pirch, L., Wressnegger, C., & Rieck, K. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.23087 · Cited by 381
Analyzing Leakage of Personally Identifiable Information in Language Models
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., & Zanella-Béguelin, S. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179300 · Cited by 552
“Research has shown the feasibility of extracting different types of information including individual sentences [9, 28], inserted canaries [8, 50, 68] as well as n -grams [42].”
BackgroundSelf-citationAdversarial machine learning : a taxonomy and terminology of attacks and mitigations
Vassilev, A., Oprea, A., Fordyce, A. J., & Anderson, H. S. (2024) · doi:10.6028/nist.ai.100-2e2023 · Cited by 317
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
Mireshghallah, F., Goyal, K., Uniyal, A., Berg-Kirkpatrick, T., & Shokri, R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.570 · Cited by 315
When Machine Unlearning Jeopardizes Privacy
Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., & Zhang, Y. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484756 · Cited by 195
“This is realistic as the target black-box model can be queried at any time, such as in the setting of MLaaS, and all of the query results can be stored locally by the adversary; this also follows the assumption of previous works [7, 59, 60, 64, 68].”
BackgroundResultWhat Does it Mean for a Language Model to Preserve Privacy?
Brown, H., Lee, K., Mireshghallah, F., Shokri, R., & Tramèr, F. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534642 · Cited by 165
“Training data extraction refers to the risk of partially extracting training samples by interacting with a trained language model [12, 13, 80, 99].”
BackgroundXu, H., Zhu, T., Zhang, L., Zhou, W., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3603620 · Cited by 164
“Recently, existing research has shown that the unlearning operation not only does not reduce the risk of user privacy leakage but actually increases this risk [106, 107].”
BackgroundCounterfactual Memorization in Neural Language Models
Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramèr, F., & Carlini, N. (2023) · neural information processing systems · doi:10.52202/075280-1708 · Cited by 284
Reconstructing Training Data with Informed Adversaries
Balle, B., Cherubin, G., & Hayes, J. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833677 · Cited by 255
Unrolling SGD: Understanding Factors Influencing Machine Unlearning
Thudi, A., Deza, G., Chandrasekaran, V., & Papernot, N. (2022) · 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp53844.2022.00027 · Cited by 135
Apruzzese, G., Anderson, H. S., Dambra, S., Freeman, D. M., Pierazzi, F., & Roundy, K. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00031 · Cited by 197
“Adversarial examples can certainly be used in malicious ways: for instance, an attacker can craft an adversarial example that evades an ML model powering a security system, e.g., a malware classifier [40].”
Hidden Backdoors in Human-Centric Language Models
Li, S., Liu, H., Dong, T., Zhao, B. Z. H., Xue, M., Zhu, H., & Lu, J. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484576 · Cited by 119
“The adversaries’ success is exacerbated by the untrustworthy supply chain and poor interpretability of such complicated large language models, further raising security concerns [3, 6, 17, 43, 44, 68].”
BackgroundAdvancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deployment
Cummings, R., Desfontaines, D., Evans, D., Geambasu, R., Huang, Y., Jagielski, M., Kairouz, P., Kamath, G., Oh, S., Ohrimenko, O., et al. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.d3197524 · Cited by 153
Self-citationAn Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models
Mireshghallah, F., Uniyal, A., Wang, T., Evans, D. M., & Berg-Kirkpatrick, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.119 · Cited by 115
“Several works have demonstrated that such large models have a high capacity for memorizing training samples during pre-training and are therefore highly susceptible to membership inference and data extraction attacks (Zanella-Béguelin et al., 2020; Carlini et al., 2021b; Nakamura et al., 2021).”
BackgroundPrivacy Issues in Large Language Models: A SurveyPreprint
Neel, S., & Chang, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.06717 · Cited by 122
Chen, C., Sun, F., Zhang, M., & Ding, B. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511997 · Cited by 110
“Consider privacy first, recent researches have shown that users’ sensitive information could be leaked from the trained models, e.g., recommender systems [50], big pretrained [4] and finetuned natu-ral language models [49].”
BackgroundPrivacy in Deep Learning: A Survey
Mireshghallah, F., Taram, M., Vepakomma, P., Singh, A., Raskar, R., & Esmaeilzadeh, H. (2020) · arXiv · doi:10.48550/arxiv.2004.12254 · Cited by 106
“inversion attack on online-learning, using a generative adversarial network and based on the difference between a model, before and after a gradient update. In the same direction, Brockschmidt et al. [57] demonstrate the information leakage of updates to language models. More recently, He et al. [47] propose a new set…”
MethodSecurity Considerations in AI-Robotics: A Survey of Current Methods, Challenges, and Opportunities
Neupane, S., Mitra, S., Fernandez, I. A., Saha, S., Mittal, S., Chen, J., Pillai, N., & Rahimi, S. (2024) · IEEE Access · doi:10.1109/access.2024.3363657 · Cited by 113
“Employing such strategy may provide guarantees in certain situations that information will not be leaked [177].”
BackgroundDP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
Du, M., Yue, X., Chow, S. S. M., Wang, T., Huang, C., & Sun, H. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3616592 · Cited by 165
“[8] define differential score and rank as two new metrics for analyzing the update leakage, enabling the recovery of new text used to update LMs.”
BackgroundTruth Serum: Poisoning Machine Learning Models to Reveal Their Secrets
Tramèr, F., Shokri, R., Joaquin, A. S., Le, H., Jagielski, M., Hong, S., & Carlini, N. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560554 · Cited by 82
“Our results imply that methods that audit privacy with average-case canaries [13, 42, 55, 63, 75] might underestimate the actual worst-case leakage under a small poisoning attack, andworstcase auditing approaches [32, 52] might more accurately measure a model’s privacy for most users.”
ResultPeng, W., Yi, J., Wu, F., Wu, S., Zhu, B. B., Lyu, L., Jiao, B., Xu, T., Sun, G., & Xie, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.423 · Cited by 122
Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defenses
Fang, H., Qiu, Y., Yu, H., Yu, W., Kong, J., Chong, B., Chen, B., Wang, X., Xia, S., & Xu, K. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-03049-x · Cited by 77
“Once obtaining the optimal z ∗ , the reconstructed images that closely align with the private images can be generated by x ∗ = G ( z ∗ ).”
Training Data Leakage Analysis in Language ModelsPreprint
Inan, H. A., Ramadan, O., Wutschitz, L., Jones, D., Rühle, V., Withers, J., & Sim, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05405 · Cited by 72
Self-citationMachine Unlearning for Random Forests
Brophy, J., & Lowd, D. (2020) · arXiv · doi:10.48550/arxiv.2009.05567 · Cited by 70
A Survey of Machine Unlearning
Nguyen, T. T., Huynh, T. T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., & Nguyen, Q. V. H. (2022) · arXiv · doi:10.48550/arxiv.2209.02299 · Cited by 193
Gyevnár, B., & Kasirzadeh, A. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01020-y · Cited by 85
“…of research focuses on the unsafe exploration of autonomous agents, primarily in the context of reinforcement learning [87], constraint violations [88], unintended behavior from incorrect domain or reward specification [81], and incorrect behavior due to continuous deployment and learning [89, 90].”
Counterfactual Memorization in Neural Language Models
Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramèr, F., & Carlini, N. (2021) · arXiv · doi:10.48550/arxiv.2112.12938 · Cited by 54
“When the training data is not available, heuristic-based methods comparing language model perplexities are used to predict whether a generation contains memorized content (Carlini et al., 2019; Thakkar et al., 2020; Thomas et al., 2020; Carlini et al., 2020; Zanella-Béguelin et al., 2020).”
MethodSoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning
Salem, A., Cherubin, G., Evans, D. M., Köpf, B., Paverd, A., Suri, A., Tople, S., & Zanella-Béguelin, S. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179281 · Cited by 102
Self-citationOlszewski, D., Lu, A., Stillman, C., Warren, K., Kitroser, C., Pascual, A., Ukirde, D., Butler, K. R. B., & Traynor, P. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3623130 · Cited by 85
Membership Inference on Word Embedding and BeyondPreprint
Mahloujifar, S., Inan, H. A., Chase, M., Ghosh, E., & Hasegawa, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11384 · Cited by 63
“Recent work has shown both successful extraction of training data (Zanella-Béguelin et al., 2020; Carlini et al., 2020; Inan et al., 2021) and membership inference (Song and Shmatikov, 2019) in language models.”
BackgroundMethodLeakage of Dataset Properties in Multi-Party Machine LearningPreprint
Zhang, W., Tople, S., & Ohrimenko, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07267 · Cited by 95
“Leakage of sensitive information about the data has been shown via membership attacks [37, 38], attribute inference attacks [12, 40], extraction of text [5] and data used in model updates [36, 42].”
MethodSelf-citationDo Language Models Plagiarize?
Lee, J., Le, T., Chen, J., & Lee, D. (2023) · Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543507.3583199 · Cited by 51
“There is a growing body of literature that aims to study the memorization of neural LMs by recovering texts in the training corpus [31, 47] or extracting artifcially injected canaries [37, 58].”
BackgroundYour Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials
Huang, Y., Li, Y., Wu, W., Zhang, J., & Lyu, M. R.-T. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660818 · Cited by 67
“…Attack aims to determine whether a target sample was used in training a target model [Shokri et al. 2017; Yeom et al. 2018], while training data extraction attack is to extract training instances by querying a trained model [Carlini et al. 2019; Salem et al. 2020; Zanella-Béguelin et al. 2020].”
MethodDo Language Models Plagiarize?
Lee, J., Le, T., Chen, J., & Lee, D. (2022) · arXiv · doi:10.48550/arxiv.2203.07618 · Cited by 51
PPA: Preference Profiling Attack Against Federated Learning
Zhou, C., Gao, Y., Fu, A., Chen, K., Dai, Z., Zhang, Z., Xue, M., & Zhang, Y. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.23171 · Cited by 47
Ganev, G., & Cristofaro, E. D. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00218 · Cited by 54
“In this section, we recap our findings, consider potential (but ineffective) countermeasures against our attacks, and conclude the paper with a few additional observations.”
Quantifying and Mitigating Privacy Risks of Contrastive Learning
He, X., & Zhang, Y. (2021) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3460120.3484571 · Cited by 66
“Besides membership inference and attribute inference, there exist a plethora of other attacks against ML models [2,4,19,22, 27, 36, 37, 40, 42, 45].”
MethodEvaluating Language Model Agency through NegotiationsPreprint
Davidson, T. R., Veselovsky, V., Josifoski, M., Peyrard, M., Bosselut, A., Kosiński, M., & West, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.04536 · Cited by 68
“First, the usual secrecy of LMs’ developers makes it impossible to ascertain that a model has not been exposed to benchmarks in their training data (Zanella-Béguelin et al., 2020).”
BackgroundZhang, Z., Wen, J., & Huang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.709 · Cited by 62
Privacy in Fine-Tuning Large Language Models: Attacks, Defenses, and Future Directions
Du, H., Liu, S., Zheng, L., Cao, Y., Nakamura, A., & Chen, L. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8183-9_25 · Cited by 34
SoK: Machine Learning GovernancePreprint
Chandrasekaran, V., Jia, H., Thudi, A., Travers, A., Yaghini, M., & Papernot, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10870 · Cited by 36
“For example, a user’s decision to revoke access to their information can itself leak information about them having been in the training dataset if it is made adaptively [301]–[303].”
BackgroundSoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
Shanmugarasa, Y., Ding, M., Chamikara, M. A. P., & Rakotoarivelo, T. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3733888 · Cited by 60
Deletion inference, reconstruction, and compliance in machine (un)learning
Gao, J., Garg, S., Mahmoody, M., & Vasudevan, P. N. (2022) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2022-0079 · Cited by 38
The mosaic memory of large language models
Shilov, I., Meeus, M., & de Montjoye, Y.-A. (2026) · Nature Communications · doi:10.1038/s41467-026-68603-0 · Cited by 36
“Following prior work, we quantify memorization of a target language model LM with tokenizer T by measuring the performance of MIAs on artificially crafted sequences, known as canaries [11, 68, 69, 82, 64, 53, 46].”
Exact and Efficient Unlearning for Large Language Model-Based Recommendation
Hu, Z., Zhang, Y., Xiao, M., Wang, W., Feng, F., & He, X. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3594687 · Cited by 37
“Some very recent studies [15], [16], [17], [18] investigate efficient unlearning techniques for information encoded in a LLM by extending traditional meth-ods [17] or utilizing in-context learning [15].”
Bayesian Estimation of Differential PrivacyPreprint
Zanella-Béguelin, S., Wutschitz, L., Tople, S., Salem, A., Rühle, V., Paverd, A., Naseri, M., Köpf, B., & Jones, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.05199 · Cited by 27
“Nonetheless, DP training with such large budgets effectively defeats attacks in many practical scenarios (Carlini et al., 2019; Jayaraman & Evans, 2019; Song & Shmatikov, 2019; Zanella-Béguelin et al., 2020).”
BackgroundSelf-citationArtificial intelligence security and privacy: a survey
He, X., Xu, G., Han, X., Wang, Q., Zhao, L., Shen, C., Lin, C., Zhao, Z., Li, Q., Yang, L., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-025-4388-5 · Cited by 37
Sanitizing Sentence Embeddings (and Labels) for Local Differential Privacy
Du, M., Yue, X., Chow, S. S. M., & Sun, H. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583512 · Cited by 62
BDMMT: Backdoor Sample Detection for Language Models Through Model Mutation Testing
Wei, J., Fan, M. Y., Jiao, W., Jin, W., & Liu, T. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3376968 · Cited by 34
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication
Luccioni, A. S., Corry, F., Sridharan, H., Ananny, M., Schultz, J., & Crawford, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533086 · Cited by 22
Membership Inference Attacks and Generalization: A Causal Perspective
Baluta, T., Shen, S., Hitarth, S., Tople, S., & Saxena, P. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560694 · Cited by 22
Self-citationA comprehensive survey and taxonomy on privacy-preserving deep learning
Tran, A.-T., Luong, T.-D., & Huynh, V.-N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127345 · Cited by 21
Privacy Leakage in Text Classification A Data Extraction Approach
Elmahdy, A., Inan, H. A., & Sim, R. (2022) · Workshop on Privacy in Natural Language Processing (PrivateNLP) · doi:10.18653/v1/2022.privatenlp-1.3 · Cited by 20
LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
Wei, C., Meng, W., Zhang, Z., Chen, M., Zhao, M., Fang, W., Wang, L., Zhang, Z., & Chen, W. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.23238 · Cited by 31
Li, H., Song, Y., & Fan, L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.429 · Cited by 33
Combing for Credentials: Active Pattern Extraction from Smart Reply
Jayaraman, B., Ghosh, E., Chase, M., Roy, S., Dai, W., & Evans, D. M. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00041 · Cited by 18
“However, unlike the threat model of canary based attacks [11], [58], where the adversary’s goal is to extract the canaries she injected, in our threat model, the adversary’s goal is to extract certain naturally occurring sensitive data from the training dataset, that is completely disjoint from the poisoning points sh…”
BackgroundMethodResultInfluentialSecurity and Privacy Challenges of AIGC in Metaverse: A Comprehensive Survey
Zhang, S., Li, H., Sun, K., Chen, H., Wang, Y., & Li, S. (2025) · ACM Computing Surveys · doi:10.1145/3729419 · Cited by 19
“DataExtraction [195] 2020 Text • Showingtheleakageoflanguagemodelsnapshotsaboutchangesinthetrainingdata.”
Towards Sentence Level Inference Attack Against Pre-trained Language Models
Gu, K., Kabir, E., Ramsurrun, N., Vosoughi, S., & Mehnaz, S. (2023) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2023-0070 · Cited by 35
“the weights of the language model with modified data [4].”
BackgroundMethodLee, T.-H., Kim, S., Lee, J., & Jun, C. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119728 · Cited by 17
The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic TextPreprint
Meeus, M., Wutschitz, L., Zanella-Béguelin, S., Tople, S., & Shokri, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.14921 · Cited by 16
“Besides their impressive fluency, LLMs are known to memorize parts of their training data (Carlini et al., 2023) and can regurgitate exact phrases, sentences, or even longer passages when prompted adversarially (Zanella-Béguelin et al., 2020; Carlini et al., 2021; Nasr et al., 2023).”
Self-citationPrivacy Budget SchedulingPreprint
Luo, T., Pan, M., Tholoniat, P., Cidon, A., Geambasu, R., & Lécuyer, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15335 · Cited by 30
“cal settings has shown that releasing multiple (versions of) ML models trained over overlapping datasets increases the attacker’s membership inference power compared to releasing just one [61].”
BackgroundUncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Frontier AI ModelsPreprint
Duan, S., Khona, M., Iyer, A., Schaeffer, R., & Fiete, I. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.14549 · Cited by 19
“However, these datasets often include proprietary, copyrighted, or otherwise private information [38, 18, 4, 10, 39, 36, 40, 46].”
BackgroundQuantifying and Analyzing Entity-Level Memorization in Large Language Models
Zhou, Z., Xiang, J., Chen, C., & Su, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29948 · Cited by 37
How to Combine Membership-Inference Attacks on Multiple Updated Machine Learning Models
Jagielski, M., Wu, S. L., Oprea, A., Ullman, J., & Geambasu, R. (2023) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2023-0078 · Cited by 14
“Some previous works have studied MI attacks involving multiple models trained on overlapping datasets [35, 44], for example arising from intermediate computations revealed by federated learning systems [29, 32, 42] or from model unlearning [9].”
BackgroundReligion and Spirituality on Social Media in the Aftermath of the Global Pandemic
Aduragba, O. T., Yu, J., & Cristea, A. I. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386754 · Cited by 16
“Language models (LMs) assign a probability to a Language models can reveal language use patterns in text [12].”
BackgroundAre GPT Embeddings Useful for Ads and Recommendation?
Peng, W., Xu, D., Xu, T., Zhang, J., & Chen, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40292-0_13 · Cited by 11
Subject-Level Membership Inference Attack via Data Augmentation and Model Discrepancy
Liu, Y., Jiang, P., & Zhu, L. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3318950 · Cited by 10
Dataset-Level Attribute Leakage in Collaborative Learning.Preprint
Zhang, W., Tople, S., & Ohrimenko, O. (2020) · arXiv (Cornell University) · Cited by 8
Self-citationA pseudonymized corpus of occupational health narratives for clinical entity recognition in Spanish
Dunstan, J., Vakili, T., Miranda, L., Villena, F., Aracena, C., Quiroga, T., Vera, P., Valenzuela, S. V., & Rocco, V. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02609-w · Cited by 7
Unintended Memorization and Timing Attacks in Named Entity Recognition Models
Ali, R. S., Zhao, B. Z. H., Asghar, H. J., Nguyen, T. T., Wood, I. D., & Kâafar, M. A. (2023) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2023-0056 · Cited by 7
The Connection between Out-of-Distribution Generalization and Privacy of ML ModelsPreprint
Mahajan, D., Tople, S., & Sharma, A. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03369 · Cited by 7
Self-citationProtecting Against Data Leakage in Federated Learning: What Approach Should You Choose?
Baracaldo, N., & Xu, R. (2022) · Springer eBooks · doi:10.1007/978-3-030-96896-0_13 · Cited by 8
Competitive artificial intelligence in information and cyber warfare
Lande, D., & Danyk, Y. (2025) · INFORMATION AND LAW · doi:10.37750/2616-6798.2025.1(52).324688 · Cited by 6
Privacy-Utility Trade-Off in Federated LLM Fine-Tuning: A Dynamic Game Approach
Wang, Y., Qian, K., Pan, Y., Su, Z., & Wang, W. (2026) · IEEE Transactions on Networking · doi:10.1109/ton.2026.3658317 · Cited by 5
“…inference threats, including memorization-based membership inference attacks [9], [10], [11], which aim to deduce the inclusion of specific samples in training, as well as gradient-based data reconstruction and extraction attacks [12], [13], [14], which attempt to recover DOs’ private data.”
Towards Privacy-Preserving Neural Architecture Search
Wang, F., Zhang, L. Y., Pan, L., Hu, S., & Doss, R. (2022) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc55528.2022.9913012 · Cited by 6
“The latest results in [10], [11] revealed that intermediate results like gradient updates leak users’ private information.”
ResultContrast-Then-Approximate: Analyzing Keyword Leakage of Generative Language Models
Zeng, Z., Xiang, T., Guo, S., He, J., Zhang, Q., Xu, G., & Zhang, T. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3392535 · Cited by 10
“The pre-training and fine-tuning process simplifies the training of LMs, and their public availability has led to several proposed attacks [9], [10], [11], [12], [13].”
BackgroundCao, Y., Ji, Y., Fu, C. Y., Dharmavaram, S., Turchioe, M. R., Benda, N. C., Mamykina, L., Sun, Y., & Xu, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791946 · Cited by 9
“Building on our findings, we call for a reconsideration of how LLMs should be evaluated and designed for healthcare.”
Memorization of Named Entities in Fine-Tuned BERT Models
Diera, A., Lell, N., Garifullina, A., & Scherp, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40837-3_16 · Cited by 6
“Initially these attacks have been designed for small LMs using academic datasets [7,62,52].”
Large Vision-Language Model Security: A Survey
Wang, T., Fang, Z., Xue, H., Zhang, C., Jin, M., Xu, W., Shu, D., Yang, S. J., Wang, Z. G., & Liu, D. (2024) · Communications in computer and information science · doi:10.1007/978-981-96-0151-6_1 · Cited by 13
Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference
Wang, S. Q., Peddinti, S. T., Taft, N., & Feamster, N. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791762 · Cited by 10
“Even benign updates to natural language models have been shown to leak changes in the training data [46], and many techniques and toolkits have been proposed to improve the success rates of these attacks [6, 7, 23, 31].”
Information Leakage from Data Updates in Machine Learning Models
Hui, T., Farokhi, F., & Ohrimenko, O. (2023) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3605764.3623905 · Cited by 3
“For example, in the data addition setting (as opposed to data change), Zanella-Béguelin et al. [15] and Jagielski et al. [9] studied how attack accuracy is affected by model update strategies and sizes of batches with new data.”
BackgroundMethodInfluentialSelf-citationSearching for Diamonds: Cross-Domain Opportunities in Cyber Threat Intelligence
Barbieri, S., Luiz Dos Santos De Souza, F., Andrey Teixeira, M., Augusto Cavalheiro Marcondes, C., & Alves Pereira, L. (2025) · IEEE Access · doi:10.1109/access.2025.3627126 · Cited by 3
“In addition, the privacy axis gains density as the risks of leaking sensitive data through unintentional memorization in models become evident, as demonstrated by Béguelin et al. [3].”
Forgetting User Preference in Recommendation Systems with Label-Flipping
Alshehri, M. A., & Zhang, X. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386603 · Cited by 4
“This is essential not only for preserving user privacy, as recommendation systems, like other neural network-based models, retain training data [1]–[3] but also for improving recommendation accuracy by eliminating noisy or irrelevant data, which is prevalent in recommendation systems [4].”
BackgroundUnraveling Model Inversion Attacks: A Survey of Machine Learning Vulnerabilities
Mostafa, T., Ibrahem, M. I., & Fouda, M. M. (2024) · International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings) · doi:10.1109/aibthings63359.2024.10863257 · Cited by 3
Sharpness-Aware Parameter Selection for Machine UnlearningPreprint
Malekmohammadi, S., Lee, H. K., & Xiong, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.06398 · Cited by 3
“Accordingly, detecting and correcting privacy issues of machine learning models has been the topic of a recent research direction (Carlini et al., 2019; Zanella-Béguelin et al., 2020; Carlini et al., 2021; Salem et al., 2018; Leino & Fredrikson, 2019; Shokri et al., 2017).”
Federated Data Analytics with Differentially Private Density Estimation Model
Wang, J., Cao, L., Chai, C., & Li, G. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00208 · Cited by 3
“…an analytical query by directly using DA to estimate the probability density of the corresponding domain values is not practical, because the estimation of DA has to leverage all local models, which inherently leaks more privacy [13], [36], [7] and introduces higher inference latency [43], [35].”
Zhang, Y., Fang, J., & Krishnakumar, A. (2026) · Prevention Science · doi:10.1007/s11121-026-01923-1 · Cited by 2
Decomposing Memorization Reduction in Privacy-Preserving Fine-Tuning of SLMs for CSIRTs
Kapelinski, C., & Kreutz, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39898-7_43 · Cited by 2
Defending against similarity shift attack for EaaS via adaptive multi-target watermarking
Yang, Z., Chen, P., Li, T., Liu, K., Huang, Y., & Lin, X. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120893 · Cited by 3
Codebreaker: Dynamic Extraction Attacks on Code Language Models
Han, C., Deng, Z., Ma, W., Zhu, X., Xue, M., Zhu, T., Wen, S., & Xiang, Y. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00124 · Cited by 2
Aliyeva, L., Abdullayev, N., Zarbiyeva, S., Huseynov, I., & Suleymanov, U. (2024) · IEEE International Conference on Application of Information and Communication Technologies (AICT) · doi:10.1109/aict61888.2024.10740413 · Cited by 2
Teleportation: Defense Against Stealing Attacks of Data-Driven Healthcare APIs
Duan, T., Wang, Z., Shen, L., Luan, S., Li, F., Doretto, G., Adjeroh, D., Niu, S., Li, J., & Tao, C. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3593470 · Cited by 1
“Earlier works explore training data extraction techniques with models relatively small in size trained on academic datasets [58], [59], while recent works pay more attention to data extraction from foundation models in real-world settings [53], [60].”
Marchant, N. G., Zhao, Y., Rubinstein, B. I. P., & Ohrimenko, O. (2025) · Springer eBooks · doi:10.1007/978-3-032-01940-0_5 · Cited by 1
Self-citationExploring the Cloud and Fog: Addressing Security Risks in AI Applications
Jain, J., & Kaur, I. (2024) · International Conference on Advancements and Key Challenges in Green Energy and Computing (AKGEC) · doi:10.1109/akgec62572.2024.10868487 · Cited by 1
Security and Privacy in AG-IoT-Enabled Unmanned Aerial Vehicles for Smart Cities
khan, Y., & Manzoor, S. (2025) · Apress eBooks · doi:10.1007/979-8-8688-1047-3_10 · Cited by 1
Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs
Li, Z., Wu, D., Wang, S., & Su, Z. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744856 · Cited by 13
“• Embedding-based Similarity (Embed): Inspired by [37–39], we implement an embedding-based method to assess semantic similarity between the extracted information and the ground truth.”
Integrity and privacy in adversarial machine learning
Jagielski, M. (2021) · Digital Repository Service (Northeastern University) · doi:10.17760/d20413920 · Cited by 1
Metadata
- Authors
- Zanella-Béguelin, S., Wutschitz, L., Tople, S., Rühle, V., Paverd, A., Ohrimenko, O., Köpf, B., & Brockschmidt, M.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- The University of Melbourne (AU)
- ORCID
- Venue
- Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2020-10-30
- Pages
- 363-375
- ISSN
- 1543-7221 (ISSN-L)
- OpenAlex
- W3027379683
- Semantic Scholar
- 782e7fba4dfa8e2095a71cfcf85995543f37ac75
- Cited by
- 86
- References
- 32
- Citations by year
- 2026: 4 · 2025: 18 · 2024: 17 · 2023: 22 · 2022: 15 · 2021: 7 · 2020: 3
- FWCI
- 4.48
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- information leakage, natural language models, model updating, differential analysis, privacy, retraining, privacy mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.