The DOI matches its DataCite record; 104 of the 112 references it cites were found in a registry or an index.
Poisoning Web-Scale Training Datasets is Practical
Open accessAbstract
Deep learning models are often trained on distributed, web-scale datasets crawled from the internet. In this paper, we introduce two new dataset poisoning attacks that intentionally introduce malicious examples to a model's performance. Our attacks are immediately practical and could, today, poison 10 popular datasets. Our first attack, split-view poisoning, exploits the mutable nature of internet content to ensure a dataset annotator's initial view of the dataset differs from the view downloaded by subsequent clients. By exploiting specific invalid trust assumptions, we show how we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD. Our second attack, frontrunning poisoning, targets web-scale datasets that periodically snapshot crowd-sourced content -- such as Wikipedia -- where an attacker only needs a time-limited window to inject malicious examples. In light of both attacks, we notify the maintainers of each affected dataset and recommended several low-overhead defenses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This oversight provides an exploitable attack surface, as malicious actors could intentionally seed misinformation into LLM training datasets through data-poisoning6 attacks that do not require direct access to model weights.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“even for manually maintained ones like Wikipedia, it is still possible for misinformation to slip in (Carlini et al., 2023).”
From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Carlini et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“pora with data in URLs risk attackers hijacking expired domains and tampering with contents (Carlini et al., 2023).”
From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Carlini et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Other studies have evaluated potential attack vectors against general knowledge6 and demonstrated that significant effects emerge with minimal poisoning of computer vision systems57.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Datasets used to train deep learning models have grown from thousands of carefully-curated examples [20, 33, 41] to webscale datasets with billions of samples automatically crawled from the internet [10, 48, 53, 57].”
BackgroundLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Datasets used to train deep learning models have grown from thousands of carefully-curated examples [20, 33, 41] to webscale datasets with billions of samples automatically crawled from the internet [10, 48, 53, 57].”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“Possibly most important among these is that uncurated data has serious implications for fairness, bias, and ethics [8, 50, 77].”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“Datasets used to train deep learning models have grown from thousands of carefully-curated examples [20, 33, 41] to webscale datasets with billions of samples automatically crawled from the internet [10, 48, 53, 57].”
BackgroundScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Deep learning is most effective when applied to large datasets [10, 30].”
BackgroundBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,058 · cited 1×
“Indeed, many language modelling datasets heavily rely on the English Wikipedia, e.g., it formed over 75% of the words in the BERT training set [25], 1.5% of the Pile dataset [27], and the entirety of the WikiText dataset [60].”
InfluentialVGGFace2: A Dataset for Recognising Faces across Pose and Age
Cao, Q., Shen, L., Xie, W., Parkhi, O. M., & Zisserman, A. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00020 · Cited by 3,032 · cited 1×
“Critically, the maintainer does not provide a snapshot of the data associated with urli, due either to untenable storage costs [3,4,17,31], privacy concerns [13, 72], or copyright limitations [16].”
Background
Research works printed but not linked: 6
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
- Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 113-125, 2019.
- Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40B: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2440-2452, 2020.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
- Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth serum: Poisoning machine learning models to reveal their secrets. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 2779-2792, 2022.
Cited by
Sorted by citations
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 242
“https://doi.org/10.1038/s41591-024-03445-1 previous studies exploring data poisoning 6,22,23 to the high-risk medical domain by examining the harm potential of practical data-poisoning attacks not requiring direct access to model weights, instead relying on misinformation uploaded to the Internet…”
InfluentialAdversarial machine learning :
Vassilev, A., Oprea, A., Fordyce, A. J., & Anderson, H. S. (2024) · doi:10.6028/nist.ai.100-2e2023 · Cited by 230
Self-citationThe Curse of Recursion: Training on Generated Data Makes Models Forget
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023) · arXiv · doi:10.48550/arxiv.2305.17493 · Cited by 151
“Recent studies have demonstrated that web-scale datasets can be poisoned by introducing malicious data into a small percentage of samples [Carlini and Terzis, 2021, Carlini et al., 2023].”
BackgroundRethinking machine unlearning for large language models
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00985-0 · Cited by 374
Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models
Shan, S., Ding, W., Passananti, J., Wu, S. L., Zheng, H.-T., & Zhao, B. Y. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00207 · Cited by 166
“This creates the possibility of data poisoning attacks [30].”
BackgroundOpen Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
“However, a similar attack is successful for instruction tuning with very few examples [Wan et al., 2023, Xu et al., 2023a], and poisoning web-scale datasets is possible under realistic assumptions [Carlini et al., 2023a].”
BackgroundOn the Risk of Misinformation Pollution with Large Language Models
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M.-Y., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.97 · Cited by 210
Medical large language models are susceptible to targeted misinformation attacks
Han, T., Nebelung, S., Khader, F., Wang, T., Müller-Franzes, G., Kuhl, C., Försch, S., Kleesiek, J., Haarburger, C., Bressem, K. K., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01282-7 · Cited by 62
Toward Federated Large Language Models: Motivations, Methods, and Future Directions
Cheng, Y., Zhang, W., Zhang, Z., Zhang, C., Wang, S., & Mao, S. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3503680 · Cited by 79
“Research has demonstrated the feasibility of poisoning expansive datasets such as LAION-400M [278], COYO-700M [279], and Wikipedia by domain purchases or crowdsourcing efforts [280].”
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57
“Recent studies have shown that web-scale datasets can be poisoned by maliciously introducing false or manipulated data into a small percentage of samples [17, 19].”
BackgroundSelf-citationCleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning
Bansal, H., Yin, F., Singhi, N., Grover, A., Yu, Y., & Chang, K.-W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00017 · Cited by 59
“Despite the successes of multimodal contrastive learning, recent studies by [6, 5] have shown that these models are vulnerable to adversarial attacks.”
BackgroundFuture Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33
Three Challenges to Secure AI Systems in the Context of AI Regulations
Hamon, R., Junklewitz, H., Garrido, J. S., & Sánchez, I. (2024) · IEEE Access · doi:10.1109/access.2024.3391021 · Cited by 36
“Other works beyond evasion attacks demonstrate that other vulnerabilities can be exploited [125], [126], [127], and that current alignment techniques may prove inefficient to protect users [128].”
BackgroundMengara, O., Avila, A. R., & Falk, T. H. (2024) · IEEE Access · doi:10.1109/access.2024.3355816 · Cited by 35
“Large datasets are vulnerable to poisoning attacks in the real world, as discussed by Carlini et al. in [113].”
BackgroundPatel, K., Shah, M., Qureshi, K. M., & Qureshi, M. R. N. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11435-z · Cited by 42
You, W., Hammoudeh, Z., & Lowd, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.833 · Cited by 35
Red teaming large language models: A comprehensive review and critical analysis
Jabbar, M. S., Al-Azani, S., Alotaibi, A., & Ahmed, M. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104239 · Cited by 31
Taylor, N., Kormilitzin, A., Lorge, I., Nevado‐Holgado, A. J., Cipriani, A., & Joyce, D. W. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102988 · Cited by 27
Training Data Extraction From Pre-trained Language Models: A Survey
Ishihara, S. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.23 · Cited by 45
Reclaiming the Digital Commons: A Public Data Trust for Training Data
Chan, A., Bradley, H., & Rajkumar, N. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604658 · Cited by 29
“[19], Carlini and Terzis [20], Li et al.”
BackgroundMethodA Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Threat Modeling AI/ML With the Attack Tree
Hoseini, S. V., Suutala, J., Partala, J., & Halunen, K. (2024) · IEEE Access · doi:10.1109/access.2024.3497011 · Cited by 22
“For example, Carlini et al. [88] disclosed two distinct attacks that assured the inclusion of harmful instances in distributed web-scaled datasets.”
Kumar, R., Prakash, R., Kumar, N., Chinara, S., & Mohammed, M. A. (2025) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-025-10436-z · Cited by 15
Ecosystem Graphs: The Social Footprint of Foundation Models
Bommasani, R., Soylu, D., Liao, T., Creel, K., & Liang, P. (2023) · doi:10.21203/rs.3.rs-2961271/v1 · Cited by 14
Large Language Models in Health Care: Charting a Path Toward Accurate, Explainable, and Secure AI
Khullar, D., Wang, X., & Wang, F. (2024) · Journal of General Internal Medicine · doi:10.1007/s11606-024-08657-2 · Cited by 15
“6 This can occur when even a small fraction (e.g., 0.1”
InfluentialAtlas: A Framework for ML Lifecycle Provenance & Transparency
Spoczynski, M., Melara, M. S., & Szyller, S. (2025) · IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw67616.2025.00058 · Cited by 15
“For example, data can be poisoned [1], [2] during collection or training.”
Machine Learning Security: Threat Model, Attacks, and Challenges
Koball, C., Wang, Y., Rimal, B. P., & Vaidyan, V. M. (2024) · Computer · doi:10.1109/mc.2024.3396357 · Cited by 11
“Carlini et al. 24 found that LLMs trained on crowd-sourced information to increase training data volume may be especially prone to poisoning attacks.”
BackgroundAgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
Chen, Z., Xiang, Z., Xiao, C., Song, D., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2407.12784 · Cited by 246
“For example, an attacker can easily inject poisoned texts by maliciously editing Wikipedia pages [4]).”
PwnPilot: Reflections on Trusting Trust in the Age of Large Language Models and AI Code Assistants
Horne, D. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00396 · Cited by 10
“A number of AI model centric studies have also identified various data poisoning risks related to this phase [22, 23, 27, 34-38].”
BackgroundRozi, M. F., Ban, T., Ozawa, S., Yamada, A., Takahashi, T., & Inoue, D. (2024) · IEEE Access · doi:10.1109/access.2024.3467180 · Cited by 11
“Using deep learning models in detection systems inevitably leads to attacks related to machine learning or artificial intelligence (AI) models, such as data poisoning [100], adversarial attacks [101], and model extraction attacks [102].”
BackgroundTrapNet: Model Inversion Defense via Trapdoor
Ma, W.-L., Wang, D.-R., Song, Y., Xue, M.-H., Wen, S., Li, Z.-D., & Xiang, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3560557 · Cited by 11
“Moreover, reliance on public datasets introduces potential risks, such as data poisoning [10], that our defense avoids by operating exclusively on private training data.”
Jiao, Y., Wang, X., & Yang, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730058 · Cited by 17
“2 ) An attacker is capable of introducing attacker-desired texts by maliciously editing Wikipedia pages, as shown in previous research [7, 104].”
Chen, Z., Gong, Y., Liu, J., Chen, M., Liu, H., Cheng, Q., Zhang, F., Lu, W. P., & Liu, X. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765023 · Cited by 6
“As established in studies [3, 18, 23], the retrieval corpus consists of millions of texts from the Internet, enabling attackers to insert maliciously crafted web content.”
Common Canvas: Open Diffusion Models Trained on Creative-Commons Images
Gokaslan, A., Cooper, A. F., Collins, J., Seguin, L., Jacobson, A., Patel, M., Frankle, J., Stephenson, C., & Kuleshov, V. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00788 · Cited by 7
Heat Death of Generative Models in Closed-Loop Learning
Marchi, M., Soatto, S., Chaudhari, P., & Tabuada, P. (2024) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc56724.2024.10886816 · Cited by 6
“One way of framing such violations is as “data poisoning” [3], a problem that is likely to become more common, as models trained from public domain internet data are especially susceptible to data poisoning attacks [6].”
Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02631-0 · Cited by 5
“The benefits, risks, and costs of replacing human activities with automated systems should be carefully evaluated, in particular from the perspective of health system resiliency and scientific accountability.”
LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks
Zhang, Q., Xiong, Z., & Mao, Z. M. (2025) · LAMPS@CCS · doi:10.1145/3733800.3763264 · Cited by 13
“These vulnerabilities provide attackers with the means to discreetly inject adversarial prompts into user clients, offering a stealthier alternative to more disruptive attacks, such as client shutdowns.”
Architectural Neural Backdoors from First Principles
Langford, H., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00060 · Cited by 5
“Note that some of these would not just be great injection points for an architectural backdoor , but also other types of backdoor attacks e.g. with domain takeover as in the Figure 8 example [Carlini et al., 2023].”
Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally
Al-Maliki, S., Qayyum, A., Ali, H. M., Abdallah, M. M., Qadir, J., Hoang, D. T., Niyato, D. T., & Al‐Fuqaha, A. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3383407 · Cited by 10
“[48] Poisoning Attack Black-box This work proposes a practical adversarial attack that can directly poison 10 well-known datasets, such as Wikipedia.”
BackgroundMethodInfluentialDetecting AI-Generated Images via CLIPPreprint
Moskowitz, A., Gaona, T., & Peterson, J. J. (2024) · Research Square · doi:10.21203/rs.3.rs-4259468/v1 · Cited by 5
Sardar, T. H., Pandey, B., & Aldasheva, L. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395730 · Cited by 5
“When asked to do so, emerging AI coding assistants will create new exploit code to known vulnerabilities [7].”
Misactivation-Aware Stealthy Backdoor Attacks on Neural Code Understanding Models
Sun, X., Xiao, Y., Bo, L., Sun, W., Liu, X., Li, B., & Zhang, J. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3614128 · Cited by 4
“We use the Spectral Signatures [14] for detecting triggers, which have also been widely used in evaluating backdoor attacks in different domains [72], [73], [74].”
Levy, M., Ravfogel, S., & Goldberg, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.569 · Cited by 5
Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
Liu, Q., Mo, W., Tong, T., Xu, J., Wang, F., Xiao, C., & Chen, M. (2024) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton63246.2024.10735305 · Cited by 12
“However, recent work [117] has shown that even a significantly smaller poison rate (0.01”
BackgroundInfluentialMithridates: Auditing and Boosting Backdoor Resistance of Machine Learning Pipelines
Bagdasarian, E., & Shmatikov, V. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690337 · Cited by 7
“An attacker who poisons a small fraction of the training data can influence what the model learns [13].”
Carroll, J. (2025) · European Conference on Cyber Warfare and Security · doi:10.34190/eccws.24.1.3713 · Cited by 4
Mo, Y., Tang, M., Lin, R., Zhou, B., & Li, X. (2025) · IEEE Access · doi:10.1109/access.2025.3582519 · Cited by 5
Multi-Agent AI Framework for Threat Mitigation and Resilience in Machine Learning Systems
Foundjem, A. T., Tidjon, L. N., Silva, L. D., & Khomh, F. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3780095 · Cited by 12
Kalyanapu, J., Dizani, F., Asher, D., Ghanbari, A., Cammarota, R., Aysu, A., & Ajorpaz, S. M. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756097 · Cited by 3
Preference Poisoning Attacks on Reward Model Learning
Wu, J., Wang, J., Xiao, C., Wang, C., Zhang, N., & Vorobeychik, Y. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00094 · Cited by 17
“Carlini et al. [9] recently shed light on this by exploiting domain-specific vulnerabilities.”
Rose, C. J., Ringsten, M., Bidonde, J., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T. (2023) · Research Square · doi:10.21203/rs.3.rs-3288515/v1 · Cited by 3
Phantom: Untargeted Poisoning Attacks on Semi-Supervised Learning
Knauer, J., Rieger, P., Fereidooni, H., & Sadeghi, A. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690369 · Cited by 2
A Rusty Link in the AI Supply Chain: Detecting Evil Configurations in Model Repositories
Ding, Z., Fu, Q., Ding, J., Deng, G., Liu, Y., & Li, Y. (2025) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw67851.2025.00036 · Cited by 2
“Suspicious Configuration Analysis is the initial step of our C ONFIG S CAN , which utilizes a rule-based approach to explore unknown configuration files by analyzing their keys [18].”
Attacking Attention of Foundation Models Disrupts Downstream Tasks
Silva, H. P., Becattini, F., & Seidenari, L. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00338 · Cited by 2
“Notably, [6] recently revealed that poisoning attacks on web-scale datasets are feasible and can significantly affect the training process of foundational models like CLIP.”
Farooq, A., Anuragi, D., Li, Z., Ziat, M., Cooperstock, J. R., & Raisamo, R. (2025) · Human-Intelligent Systems Integration · doi:10.1007/s42454-025-00085-9 · Cited by 2
“Validating the provenance and integrity of documents before ingestion into the local vectorDB is critical to prevent data poisoning; check-sums or digital signatures could be employed (Carlini et al. 2024).”
Jędrasiak, K. (2025) · Bezpieczeństwo i Technika Pożarnicza · doi:10.12845/sft.66.2.2025.10 · Cited by 2
A Survey on Deep Learning for Face Anti-Spoofing in Automotive Applications
Ruiz, M., Hassani, H., & Nikan, S. (2026) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2026.3669032 · Cited by 2
Safety and security of large language models in healthcare
Clusmann, J., Freyer, O., Ostermann, M., Ferber, D., Laleh, N. G., Hilgers, L., Kolbinger, F. R., Schneider, C. V., Downing, A., Wekenborg, M., et al. (2026) · Nature · doi:10.1038/s41586-026-10687-1 · Cited by 5
“Developers should routinely report risks 70 and adopt stringent model-selection protocols 69,71 to ensure that deployed LLMs originate exclusively from trusted and authenticated sources.”
Tan, M., Yamaguchi, K., Raney, A., Nockles, V., Leblanc, M., & Bendelac, S. (2024) · Assurance and Security for AI-enabled Systems · doi:10.1117/12.3021908 · Cited by 4
“83 The attackers are not expected to know the model architectures that will be used for training, hence they are likely to have black-box knowledge.”
BackgroundAdversarially Robust Anti-Backdoor Learning
Zhao, Q., & Wressnegger, C. (2024) · doi:10.1145/3689932.3694763 · Cited by 2
“Backdoors are injected by either model manipulation [1, 4, 19, 63] or dataset poisoning [5, 11, 28, 45], where the latter is most commonly used due to the simplicity and practicality [7].”
Integrating Generative AI in Cybersecurity Curricula
AlOmar, B., & Trabelsi, Z. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016426 · Cited by 2
“Backdoor Implementation via Malicious Serialization embeds dormant triggers in serialized data structures, remaining inactive during normal operations but activating under specific conditions to execute malicious behaviors [59].”
Generalized Adversarial Code-Suggestions: Exploiting Contexts of LLM-based Code-Completion
Rubel, K., Noppel, M., & Wressnegger, C. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3733870 · Cited by 3
“The real-world costs for poisoning the training data, in turn, depend on the setting [14].”
Model-agnostic clean-label backdoor mitigation in cybersecurity environments
Severi, G., Boboila, S., Holodnak, J. T., Kratkiewicz, K., Izmailov, R., Lucia, M. D., & Oprea, A. (2025) · IEEE Military Communications Conference (MILCOM) · doi:10.1109/milcom64451.2025.11310745 · Cited by 3
Towards the Effect of Examples on In‐Context Learning: A Theoretical Case Study
He, P., Cui, Y., Xu, H., Liu, H., Yamada, M., Tang, J., & Xing, Y. (2025) · Stat · doi:10.1002/sta4.70045 · Cited by 3
“In real practice, it is often observed that the real data contradict the knowledge from the pre-training data, or the quality is not perfect (Carlini et al. 2024; He et al. 2024).”
Anichkov, Y., Popov, V., & Bolovtsov, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-80853-1_31 · Cited by 2
Lee, C., Kao, C.-C., Yeh, C.-H., Lu, C.-S., Yu, C.-M., & Chen, C.-S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00630 · Cited by 1
“Several studies [8, 41, 55] have explored the potential threat posed by uncurated data in collection pipelines.”
Dual-Model Backdoor Defense with Contrast Learning
Chen, Y., & Yin, J.-L. (2025) · International Conference on Big Data and Privacy Computing (BDPC) · doi:10.1109/bdpc63545.2025.11135804 · Cited by 1
“Fine-tuning (FT) methods have been used to improve model robustness to backdoor attacks [10], [14], [17], but they have not yet been fully evaluated in a variety of attack setups, especially in the more practical regime of low poisoning rates [18], [19].”
GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Search
Ben-Tov, M., & Sharif, M. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765095 · Cited by 1
“A fundamental risk in search systems, and specifically in embedding-based retrieval, is Search-Engine Optimization (SEO), seeking to promote potentially malicious content’s ranking for certain target query distributions [71].”
InfluentialLS 2 : Boosting Hidden Separation for Backdoor Defense with Learning Speed-driven Label Smoothing
Peng, J., Yang, H., He, H., Zhao, J., He, H., Dong, H., & Zhang, W. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3616621 · Cited by 1
“Moreover, Carlini et al. [2] recently showed that modifying a small fraction of web-scale datasets is su ffi cient to implant a backdoor into the victim model.”
Push and Pull : Defending against Retrieval Poisoning Attacks via Embedding Space Reshaping
He, L., Zhang, X., Liu, Q., Li, C., Liu, Z., Zhou, P., & Su, S. (2026) · ACM Transactions on Information Systems · doi:10.1145/3800948 · Cited by 2
Mitigating Cognitive Vulnerabilities in Code Generation via Multi-Agent Adversarial Debate
Liu, S., Guo, Q., Liu, X., & Liu, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792557 · Cited by 1
“]; (ii) deficient counterfactual reasoning , which limits the model’s ability to anticipate rare or adversarial scenarios [16]; and (iii) susceptibility to adversarial manipulation , where carefully crafted prompts can induce incorrect or even malicious outputs [3, 28, 31].”
Exact Certification of Data-Poisoning Attacks Using Mixed-Integer Programming
Sosnin, P., Knapp, J., Kennedy, F., Collyer, J., & Tsay, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-27242-3_32 · Cited by 1
Bias in AI: a framework for understanding the sources of bias in artificial intelligence
Robinig, W., & Wallner, J. P. (2026) · AI and Ethics · doi:10.1007/s43681-026-01283-0 · Cited by 1
“Such deliberate manipulation typically enters either through the data (poisoning a training data corpus) [ 4] or through the interaction layer (steering outputs with crafted prompts or injected content) [ 1 ], and can therefore also be localized within this framework.”
Guo, W., Ma, X., Ge, P., Chen, Y., Yue, Q., & Zhang, Y. (2024) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics62450.2024.00097 · Cited by 1
“However, the security of the dataset collected in this way cannot be guaranteed because it has been shown that poisoning a web-scale dataset is possible [21], but due to the large size of the dataset, the user cannot check the dataset one by one, which allows the attacker to exploit the dataset.”
BackgroundClean Label Attacks Against SLU Systems
Xinyuan, H. L., Joshi, S., Thebaud, T., Villalba, J., Dehak, N., & Khudanpur, S. P. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832144 · Cited by 3
“These attacks have emerged as a realistic and effective threat model as the training of large models [8, 9] increasingly necessitates sourcing large amounts of data from the Internet, some of which might be vulnerable to manipulation by an adversary [10].”
PACOL: Poisoning Attacks Against Continual LearnersPreprint
Ditzler, G., & Li, H. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4517547 · Cited by 3
Campbell, R. (2026) · Systems · doi:10.3390/systems14050593 · Cited by 3
Bespoke Large Language Models for Digital Triage Assistance in Mental Health CarePreprint
Taylor, N., Kormilitzin, A., Lorge, I., Nevado‐Holgado, A. J., & Joyce, D. W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4777351
SINTESIS TEKS KE GAMBAR: TINJAUAN ATAS DATASET
Arifin, S. R. (2024) · Jurnal EEICT (Electric Electronic Instrumentation Control Telecommunication) · doi:10.31602/eeict.v7i1.13066 · Cited by 1
“Diperlukan kurasi dan filtering yang hati-hati [71] (Srinivasan et al., 2022).”
BackgroundAgent Deprogramming Finetuning Away Backdoor Triggers for Secure Machine Learning Models/LLMs
Edwards, D., & Rawat, D. B. (2024) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn61486.2024.10637573 · Cited by 2
“Previous work explored introducing backdoors through data poisoning with crafted input samples [1]–[4] or in outsourced training and model reuse scenarios [1], [3]–[5].”
BackgroundInfluentialFCert: Certifiably Robust Few-Shot Classification in the Era of Foundation Models
Wang, Y., Zou, W., & Jia, J. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00240 · Cited by 1
Restoring Unintended Model Learning by Error Correcting Code
Ashizawa, N., Shibahara, T., Kiribuchi, N., Saisho, O., & Yanai, N. (2024) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw65225.2024.00013
“Such unreliable data may cause unintended adverse effects on learning [2], [3].”
A Survey on Emoji and Unicode-Based Data Masking Attacks on AI Systems
Sakpal, A. (2025) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2025.72523
SESP: Semantic Entropy-based Self-play Security Alignment Mechanism for Large Language ModelsPreprint
Sun, H., Cao, L., Zhao, Z., Du, X., Liu, A., Sang, W., & Li, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5332865
Data Leakage and Data Poisoning
Kobayashi, M. (2025) · Springer eBooks · doi:10.1007/978-981-96-9223-1_5 · Cited by 1
Exploiting Contexts of LLM-based Code-Completion
Noppel, M., Rubel, K., & Wressnegger, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02813-6_26
Neretin, O., & Kharchenko, V. S. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-05802-7_34
Large Language Models as Search Engines: Societal Challenges
Sadeddine, Z., Maxwell, W., Varoquaux, G. P., & Suchanek, F. M. (2025) · ACM SIGIR Forum · doi:10.1145/3769733.3769746 · Cited by 1
“However, training data can be not only naturally bad, but also purposefully poisoned: Microsoft’s chatbot Tay was shutdown after only 16 hours online because it started posting inflammatory and offensive tweets, caused by its interactions with troll users [Victor, 2016].”
On the Resilience of Traditional AI Algorithms Toward Poisoning Attacks for Vulnerability Detection
González‐Manzano, L., & García-Alfaro, J. (2025) · IET Information Security · doi:10.1049/ise2/9997989
“For example, multilingual Wikipedia editions have shown vulnerabilities to poisoning rates up to 25.3 % , demonstrating that even well-curated sources can be signi fi cantly compromised [52].”
Influential109PwnPilot: could an adversary be pair programming with our most trusted software engineers?
Horne, D. (2025) · De Gruyter eBooks · doi:10.1515/9783111436548-006
A Practical and Secure Byzantine Robust Aggregator
Lee, D. Z., Kolluri, A., Saxena, P., & Chang, E. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744851 · Cited by 2
VillainNet: Targeted Poisoning Attacks Against SuperNets Along the Accuracy-Latency Pareto Frontier
Oygenblik, D., Vemulapalli, A. S., Agrawal, A., Sanyal, D., Tumanov, A., & Saltaformaggio, B. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765185
“As with conventional AI systems, SuperNets must contend with adversarial manipulations such as data poisoning and backdoor attacks [13, 48, 39, 17, 12].”
CatPoison: Category-Oriented Knowledge Poisoning Attacks in Retrieval-Augmented Generation Systems
Mei, H., Du, Z., Cai, C., Cheng, C., & Chen, J. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00254
“Automated fact-checking, cross-source validation, and embedding-based anomaly detection can help identify poisoning texts targeting an entire semantic category [6], [17].”
CAN LARGE LANGUAGE MODELS ACT AS “CO-AUDITORS”?
Emekci, H. (2026) · Denetişim · doi:10.58348/denetisim.1782835
Secure Prompt Engineering Patterns for Cloud LLM Agents
Sudarsan, S., Patel, A., Upadhyay, C., Gudur, V., & Matam, P. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395764 · Cited by 1
“These threats exploit LLMs' natural language interfaces to mislead them into executing unauthorized or malicious operations in cloud environments [27].”
(2025) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83549-238-320251010
Large Language Models as Search Engines: Societal Challenges
Sadeddine, Z., Maxwell, W., Varoquaux, G. P., & Suchanek, F. M. (2025) · HAL · doi:10.1145/3769733.376974
Privatar: A Privacy-Preserving Framework for LLM Agents
Jian, R., Zhu, Z., Shang, Y., Shen, Y., & Liu, K. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00011
“Alternatively, they may actively attack LLMs to extract private user data; for instance, tainted training data may be disseminated over the internet [18], [19], subsequently integrated into LLM training pipelines, thereby compromising model security.”
Tiny Poison, Big Effect: Comprehensive Multi-Vector Poisoning Assessment of Small Language Models
Sharma, N., & Singh, U. (2026) · International Conference on Computing, Communication, Control and Cyber-Physical Systems (I5CPS) · doi:10.1109/i5cps67958.2026.11452345
“Carlini et al. [6] systematically studied deduplication, finding it reduces memorization but doesn’t eliminate backdoors.”
Influential
Metadata
- Authors
- Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- ETH Zurich (CH)
- Nvidia (United Kingdom) (GB)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 407-425
- ISSN
- 1081-6011 (ISSN-L)
- arXiv
- 2302.10149
- OpenAlex
- W4402264526
- Semantic Scholar
- 2cf43a61d0937ad25f23eaef7c90253ab799b3c7
- Cited by
- 138
- References
- 112
- Citations by year
- 2026: 53 · 2025: 44 · 2024: 29 · 2023: 12
- FWCI
- 37.13
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- deep learning security, Wikipedia, crowdsourced data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
