The DOI matches its DataCite record; 104 of the 112 references it cites were found in a registry or an index.
Poisoning Web-Scale Training Datasets is Practical
Open accessAbstract
Deep learning models are often trained on distributed, web-scale datasets crawled from the internet. In this paper, we introduce two new dataset poisoning attacks that intentionally introduce malicious examples to a model's performance. Our attacks are immediately practical and could, today, poison 10 popular datasets. Our first attack, split-view poisoning, exploits the mutable nature of internet content to ensure a dataset annotator's initial view of the dataset differs from the view downloaded by subsequent clients. By exploiting specific invalid trust assumptions, we show how we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD. Our second attack, frontrunning poisoning, targets web-scale datasets that periodically snapshot crowd-sourced content -- such as Wikipedia -- where an attacker only needs a time-limited window to inject malicious examples. In light of both attacks, we notify the maintainers of each affected dataset and recommended several low-overhead defenses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This oversight provides an exploitable attack surface, as malicious actors could intentionally seed misinformation into LLM training datasets through data-poisoning6 attacks that do not require direct access to model weights.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“even for manually maintained ones like Wikipedia, it is still possible for misinformation to slip in (Carlini et al., 2023).”
From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Carlini et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“pora with data in URLs risk attackers hijacking expired domains and tampering with contents (Carlini et al., 2023).”
From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Carlini et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Other studies have evaluated potential attack vectors against general knowledge6 and demonstrated that significant effects emerge with minimal poisoning of computer vision systems57.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Datasets used to train deep learning models have grown from thousands of carefully-curated examples [20, 33, 41] to webscale datasets with billions of samples automatically crawled from the internet [10, 48, 53, 57].”
BackgroundLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Datasets used to train deep learning models have grown from thousands of carefully-curated examples [20, 33, 41] to webscale datasets with billions of samples automatically crawled from the internet [10, 48, 53, 57].”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 2×
“Possibly most important among these is that uncurated data has serious implications for fairness, bias, and ethics [8, 50, 77].”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“Datasets used to train deep learning models have grown from thousands of carefully-curated examples [20, 33, 41] to webscale datasets with billions of samples automatically crawled from the internet [10, 48, 53, 57].”
BackgroundScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Deep learning is most effective when applied to large datasets [10, 30].”
BackgroundBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,058 · cited 1×
“Indeed, many language modelling datasets heavily rely on the English Wikipedia, e.g., it formed over 75% of the words in the BERT training set [25], 1.5% of the Pile dataset [27], and the entirety of the WikiText dataset [60].”
InfluentialVGGFace2: A Dataset for Recognising Faces across Pose and Age
Cao, Q., Shen, L., Xie, W., Parkhi, O. M., & Zisserman, A. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00020 · Cited by 3,032 · cited 1×
“Critically, the maintainer does not provide a snapshot of the data associated with urli, due either to untenable storage costs [3,4,17,31], privacy concerns [13, 72], or copyright limitations [16].”
Background
Research works printed but not linked: 6
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
- Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 113-125, 2019.
- Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40B: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2440-2452, 2020.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
- Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth serum: Poisoning machine learning models to reveal their secrets. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 2779-2792, 2022.
Cited by
Sorted by citations
Items 101–138 of 138
PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Reading
Wu, Y., Liu, X., Feng, Y., Ding, J., & Ma, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792597 · Cited by 2
“Attackers exploit this by injecting poisoned text into publicly editable sources [4].”
CACTEE: Confidential Asset Certification using Trusted Execution Environments
Akkuş, İ. E., & Rimac, I. (2026) · Workshop on System Software for Trusted Execution (SysTEX) · doi:10.1145/3805690.3805734
Revisiting Backdoor Threat in Federated Instruction Tuning From a Signal Aggregation Perspective
Zhao, H., Hu, J., & Liu, G. H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462088
BadLLM-TG: A Backdoor Defender Powered by LLM Trigger Generator
Zhang, R., Gao, H., Jian, S., Tan, Y., & Zhou, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463345
“Backdoor attacks can severely compromise models with only minimal perturbations [11,18], highlighting the urgent need for effective defense mechanisms.”
Covert Knowledge Poisoning Attacks in Retrieval-Augmented Code Generation
Zhang, Y., Chen, M., He, X., Cong, T., Xu, K., & Li, Q. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3686845
“Adversaries can poison the knowledge base by injecting malicious code into public repositories—a threat recently demonstrated in prior work [8], [9].”
InfluentialExplainability-Guided Untargeted Attacks on Knowledge Graph Embedding
Lin, Y., Yu, H., Liang, K., Wang, M., Yang, L., & Liu, X. Y. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3689651 · Cited by 1
“Although KGs receive remarkable achievements, they are also vulnerable to adversarial attacks [13], [14], [15], [16], [17], [18].”
Security as an Ethical Imperative in AI Systems
Andrade, R., Ayala, C., & Morillo, P. (2026) · Synthesis lectures on computer science · doi:10.1007/978-3-032-25789-5_5
Machine Learning and Hardware Security: The Role of AI for Hardware in the Security Era
Deng, S., Xu, Y., Li, M., Jin, Y., & Wang, J. (2026) · Springer eBooks · doi:10.1007/978-3-032-17948-7_12
Mitigating Security Risks in Large Language Models: A Full Lifecycle Perspective
Wang, Y. K., Bai, Z., Huo, J., Wang, B., Liu, J., Cao, H., Meng, F., Yang, X., & Gao, Y. (2026) · Machine Learning · doi:10.1007/s10994-025-06930-x
“Poison - ing attacks can target pre-training datasets (Carlini et al., 2024), in-context learning (ICL) (He et al., 2024 ), instruction fine-tuning (Jiang et al., 2024; Qi et al., 2024; Wan et al., 2023; Shu et al., 2023; Qiang et al., 2024 ), and RLHF (Wang et al., 2024 ; Pathmanathan et al.,…”
From AI-generated content to agentic action: Security and safety threats in generative AI
Zhang, Z., Li, Q., Cao, J., Liu, L., & Ni, J. (2026) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2026.05.002 · Cited by 1
“For example, split-view attacks can poison 0.01% of LAION-400M for approximately $60 [18].”
InfluentialSide-Channel Privacy Attacks on Energy-Efficient Mobile Agents: A Membership Inference Perspective
Ehsan, I., Niu, J., & Zhang, Y. (2026) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3812836.3814743
Zheng, Z., Choi, Y., Huang, C., Cao, H., Tian, S., & Pei, T. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3699202
“As reported in [38], a conservative estimate puts the probability of malicious edits to Wikipedia documents at 6.5%.”
InfluentialXu, Z., Gardiner, J., & Belguith, S. (2026) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3779208.3785271
“Despite extensive data filtering during training, traces of harmful content can still persist [9, 43].”
A Formal Security Framework for Model Context Protocol-Based Tool Access inAgentic AI SystemsPreprint
Alam, M. Z. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6891368
Infrastructure as Compromise: Abusing Residual Trust in Infrastructure as Code Tools
Yang, R., Chaiwut, N., & Nikiforakis, N. (2026) · ACM Conference on Data and Application Security and Privacy (CODASPY) · doi:10.1145/3800506.3803500
Berini, A. D. E., Jamil, N., Lakas, A., Doshmangir, L., Ferrag, M. A., & Lam, K. (2026) · ACM International Workshop on Security and Privacy Analytics (IWSPA) · doi:10.1145/3806007.3810962
Liu, S., & Yang, X. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-9884-7_3
Evaluating Design Decisions and Bias Resistance for Passive DNS-Based Domain Rankings
Pochat, V. L., Fernandez, S., Tajalizadehkhoob, S., Desmet, L., Duda, A., Joosen, W., & Korczyński, M. (2026) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2026.3705306
“…graph-based designs by artificially increasing the number of links between their website and the rest of the web: outgoing links can be added to their websites, and incoming backlinks to their websites can be inserted on popular public platforms such as blogs, social networks, wikis and forums [61].”
Liu, C., Qian, L., Jadhav, C. D., & Zhao, J. Z. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571647
“For instance, adversaries can manipulate the retrieved content by injecting poisoned documents into the local knowledge base [9].”
Detecting Sentiment Steering Attacks on RAG-enabled Large Language Models
Mohan, A., Mahadik, S. S., Mukherjee, M., Pawar, P. M., Muthalagu, R., & Lloret, J. (2026) · International Conference on Communications · doi:10.1109/icc59461.2026.11588193
“Therefore, the development of a security model in the IoT context inspires the research.”
Fairness in vision-language text-to-image retrieval: Natural biases and adversarial threats
Lazzaro, D., Ciná, A. E., Roli, F., Anguita, D., Biggio, B., Laurita, G., & Oneto, L. (2026) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2026.111385
Affective Computing in the Age of GenAI: Balancing Risks and Rewards
Latif, S., Shoukat, M., Pervez, F., Shamshad, F., Usama, M., Sandhu, M., & Schuller, B. W. (2026) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2026.3678879
“Carlini et al. [106] demonstrated that adversarial perturbations in training data can cause models to misclassify or behave unexpectedly.”
!Imperio, smolVLA: The Implications of Data Poisoning on Open Source Robotics
Bühler, S., & Schutera, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-32335-4_2
“Data poisoning and backdoor attacks are studied extensively in supervised learning and Large Language Models (LLMs) [1, 3, 4, 9, 12, 14, 17].”
MC-PoisonVerif: Model Checking-Driven Poisoning Attack Formal Verification Algorithm
Lv, C., Song, Y., Li, Y., Zhou, W., Chenyu, G., & Zhang, X. (2026) · Mathematics · doi:10.3390/math14142617
Kakebayashi, R., & Mori, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31441-3_7
Reproducibility Is the New Copyleft: Defining AGI-Oriented Reproducible Builds
Hatta, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33010-9_25
RefineRAG: Word-Level Poisoning Attacks via Retriever-Guided Text Refinement
Wang, Z., Wang, G., & Wang, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31583-0_35
DidData: A Trust-Aware Protocol for Sovereign AI Data Supply Chains
Liu, Y., Cheng, T., Yin, Z., & Liu, Y. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00105
Data security in large language models: risks, defense, and directions
Chen, K., Zhou, X., Yu, Y., Lin, Y., Chen, H., Cai, C., & Shen, L. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01200-9 · Cited by 2
Gao, Q., Gao, Y., Kang, H., Jiang, L., Zhang, X., & Li, Z. (2026) · International Conference on Electronics Technology (ICET) · doi:10.1109/icet69987.2026.11659295
“While traditional data poisoning targets training sets [13], recent threats expand to instruction tuning and multi-agent communication.”
Counterfactual incremental defense against poisoning attacks
Wang, H., & Kandah, F. I. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116930
Training on Clean Data but Getting Backdoored Models! A Poisoning Attack on Code Encoders
Xiao, Y., Liu, X., Yang, Z., Bo, L., & Sun, X. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773236
SoK: On the Survivability of Backdoor Attacks on Unconstrained Face Recognition Systems
Roux, Q. L., Teglia, Y., Furon, T., Loubet-Moundi, P., & Bourbao, E. (2026) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml68715.2026.00068 · Cited by 1
“A common injection method is targeted data poisoning [25], where a fraction of a training dataset is modified to contain the malicious trigger patterns.”
Strategic Oversight for AI Safety: A Stackelberg Security Games Perspective
Kim, C. W., Choo, D., Neoh, T. Y., & Tambe, M. (2026) · IEEE Intelligent Systems · doi:10.1109/mis.2026.3725453
“18,29,52 Failure Modes During Training Data poisoning has been extensively studied for both pretraining 14,48 and posttraining stages.”
Ge, H., Shi, Z., & Nie, Y. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-5712-6_34
Perturb and Recover: Fine-Tuning for Effective Backdoor Removal from CLIP
Singh, N. D., Croce, F., & Hein, M. (2026) · doi:10.1109/cvprf71909.2026.00592
“For multi-modal models like CLIP trained on massive amounts of web-scraped data, it is possible to manipulate training data as demonstrated by [7] who got control of a fraction of LAION-400M [39] or Coyo-700M [5] at low cost.”
InfluentialAjibola, T., & Okpiyalele, K. (2026) · International Journal of Scientific Research in Artificial Intelligence and Machine Learning · doi:10.32628/ijsraiml2624121
RAGtrap: Source Revocation and Indexed Provenance Lookup for Poisoned RAG Corpora
Kapelinski, C., & Kreutz, D. (2026) · Simpósio Brasileiro de Cibersegurança (SBSeg) · doi:10.5753/sbseg_estendido.2026.29850
Metadata
- Authors
- Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- ETH Zurich (CH)
- Nvidia (United Kingdom) (GB)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 407-425
- ISSN
- 1081-6011 (ISSN-L)
- arXiv
- 2302.10149
- OpenAlex
- W4402264526
- Semantic Scholar
- 2cf43a61d0937ad25f23eaef7c90253ab799b3c7
- Cited by
- 138
- References
- 112
- Citations by year
- 2026: 53 · 2025: 44 · 2024: 29 · 2023: 12
- FWCI
- 37.13
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- deep learning security, Wikipedia, crowdsourced data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
