Models Are Codes: Towards Measuring Malicious Code Poisoning Attacks on Pre-trained Model Hubs
Open accessAbstract
The proliferation of pre-trained models (PTMs) and datasets has led to the emergence of centralized model hubs like Hugging Face, which facilitate collaborative development and reuse. However, recent security reports have uncovered vulnerabilities and instances of malicious attacks within these platforms, highlighting growing security concerns. This paper presents the first systematic study of malicious code poisoning attacks on pre-trained model hubs, focusing on the Hugging Face platform. We conduct a comprehensive threat analysis, develop a taxonomy of model formats, and perform root cause analysis of vulnerable formats. While existing tools like Fickling and ModelScan offer some protection, they face limitations in semantic-level analysis and comprehensive threat detection. To address these challenges, we propose MalHug, an end-to-end pipeline tailored for Hugging Face that combines dataset loading script extraction, model deserialization, in-depth taint analysis, and heuristic pattern matching to detect and classify malicious code poisoning attacks in datasets and models. In collaboration with Ant Group, a leading financial technology company, we have implemented and deployed MalHug on a mirrored Hugging Face instance within their infrastructure, where it has been operational for over three months. During this period, MalHug has monitored more than 705K models and 176K datasets, uncovering 91 malicious models and 9 malicious dataset loading scripts. These findings reveal a range of security threats, including reverse shell, browser credential theft, and system reconnaissance. This work not only bridges a critical gap in understanding the security of the PTM supply chain but also provides a practical, industry-tested solution for enhancing the security of pre-trained model hubs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Data poisoning allows adversaries to inject manipulated samples into training datasets to bias model behavior.
“Data poisoning allows adversaries to inject manipulated samples into training datasets to bias model behavior [19,20].”
From AI Supply Chain Security: MBOM-PQC Provenance, PQC Attestation, and a Maturity Model for Quantum-Resistant Assurance (2026) · cites this paper as [19]
Section 2.2 describes malicious scripts embedded in datasets, and the Introduction describes harmful code injected into model files. Neither passage addresses manipulated training samples used to bias model behavior. Searches for “data poisoning” and “training datasets manipulated samples” found no passage in our copy establishing that claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SoK: Taxonomy of Attacks on Open-Source Software Supply Chains
Ladisa, P., Plate, H., Martinez, M., & Barais, O. (2023) · 2023 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46215.2023.10179304 · Cited by 208 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- Ruian Duan, Omar Alrawi, Ranjita Pai Kasturi, Ryan Elder, Brendan Saltaformag- gio, and Wenke Lee. 2021. Towards Measuring Supply Chain Attacks on Package Managers for Interpreted Languages. In 28th Annual Network and Distributed System Security Symposium, NDSS. https://www.ndss-symposium.org/wp- content/uploads/ndss2021_1B-1_23055_paper.pdf
- William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research 23, 120 (2022), 1-39. http://jmlr.org/papers/v23/21- 0998.html
- Wenbo Guo, Zhengzi Xu, Chengwei Liu, Cheng Huang, Yong Fang, and Yang Liu. 2023. An Empirical Study of Malicious Code In PyPI Ecosystem. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 166-177.
- Jiayi Hua, Kailong Wang, Meizhen Wang, Guangdong Bai, Xiapu Luo, and Haoyu Wang. 2024. MalModel: Hiding Malicious Payload in Mobile Deep Learning Models with Black-box Backdoor Attack. arXiv preprint arXiv:2401.02659 (2024). · arXiv
- Cheng Huang, Nannan Wang, Ziyan Wang, Siqi Sun, Lingzi Li, Junren Chen, Qianchong Zhao, Jiaxuan Han, Zhen Yang, and Lei Shi. 2024. DONAPI: Malicious NPM Packages Detector using Behavior Sequence Knowledge Mapping. arXiv preprint arXiv:2403.08334 (2024). · arXiv
- Wenxin Jiang, Chingwo Cheung, George K Thiruvathukal, and James C Davis. 2023. Exploring naming conventions (and defects) of pre-trained deep learning models in hugging face and other model hubs. arXiv preprint arXiv:2310.01642 (2023). · arXiv
- Wenxin Jiang, Nicholas Synovic, Matt Hyatt, Taylor R. Schorlemmer, Rohan Sethi, Yung-Hsiang Lu, George K. Thiruvathukal, and James C. Davis. 2023. An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE '23). IEEE Press, 2463-2475. https://doi.org/10.1109/ICSE48619.2023.00206 · DOI · Paper page
- Wenxin Jiang, Nicholas Synovic, Rohan Sethi, Aryan Indarapu, Matt Hyatt, Tay- lor R. Schorlemmer, George K. Thiruvathukal, and James C. Davis. 2022. An Empirical Study of Artifacts and Security Risks in the Pre-trained Model Supply Chain. In Proceedings of the 2022 ACM Workshop on Software Supply Chain Of- fensive Research and Ecosystem Defenses (Los Angeles, CA, USA) (SCORED'22). Association for Computing Machinery, New York, NY, USA, 105-114. https: //doi.org/10.1145/3560835.3564547 · DOI
- Wenxin Jiang, Jerin Yasmin, Jason Jones, Nicholas Synovic, Jiashen Kuo, Nathaniel Bielanski, Yuan Tian, George K. Thiruvathukal, and James C. Davis. 2024. PeaT- MOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software. In Proceedings of the 21st International Conference on Mining Software Repositories (Lisbon, Portugal) (MSR '24). Association for Computing Machinery, New York, NY, USA, 431-443. https://doi.org/10.1145/3643991.3644907 · DOI
- Li Li, Jiawei Wang, and Haowei Quan. 2022. Scalpel: The Python Static Analysis Framework. arXiv preprint arXiv:2202.11840 (2022). · arXiv
- Ningke Li, Shenao Wang, Mingxi Feng, Kailong Wang, Meizhen Wang, and Haoyu Wang. 2023. MalWuKong: Towards Fast, Accurate, and Multilingual Detection of Malicious Code Poisoning in OSS Supply Chains. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1993-2005. https://doi.org/10.1109/ASE56229.2023.00073 · DOI
- Yuanchun Li, Jiayi Hua, Haoyu Wang, Chunyang Chen, and Yunxin Liu. 2021. Deeppayload: Black-box backdoor attack on deep learning models through neural payload injection. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 263-274.
- Xiangru Lian, Binhang Yuan, Xuefeng Zhu, Yulong Wang, Yongjun He, Honghuan Wu, Lei Sun, Haodong Lyu, Chengjun Liu, Xing Dong, Yiqiao Liao, Mingnan Luo, Congfei Zhang, Jingru Xie, Haonan Li, Lei Chen, Renjie Huang, Jianying Lin, Chengchun Shu, Xuezhong Qiu, Zhishan Liu, Dongying Kong, Lei Yuan, Hai Yu, Sen Yang, Ce Zhang, and Ji Liu. 2022. Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Washington DC, USA) (KDD '22). Association for Computing Machinery, New York, NY, USA, 3288-3298. https://doi.org/10.1145/3534678.3539070 · DOI
- Michael M McKerns, Leif Strand, Tim Sullivan, Alta Fang, and Michael AG Aivazis. 2012. Building a framework for predictive science. arXiv preprint arXiv:1202.1056 (2012). · arXiv
- Marc Ohm, Henrik Plate, Arnold Sykosch, and Michael Meier. 2020. Backstab- ber's Knife Collection: A Review of Open Source Software Supply Chain Attacks. In Detection of Intrusions and Malware, and Vulnerability Assessment, Clémen- tine Maurice, Leyla Bilge, Gianluca Stringhini, and Nuno Neves (Eds.). Springer International Publishing, Cham, 23-43.
- Shenao Wang, Yanjie Zhao, Xinyi Hou, and Haoyu Wang. 2024. Large language model supply chain: A research agenda. arXiv preprint arXiv:2404.12736 (2024). · arXiv
- X. Zhang, Z. Zhang, S. Ji, and T. Wang. 2021. Trojaning Language Models for Fun and Profit. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE Computer Society, Los Alamitos, CA, USA, 179-197. https: //doi.org/10.1109/EuroSP51992.2021.00022 · DOI
Cited by
Sorted by citations
Campbell, R. (2026) · Systems · doi:10.3390/systems14050593 · Cited by 3 · see citing works
- Authors
- Zhao, J., Wang, S., Zhao, Y., Hou, X., Wang, K., Gao, P., Zhang, Y., Wei, C., & Wang, H.
- Affiliations
- Huazhong University of Science and Technology (CN)
- Ant Group (China) (CN)
- ORCID
- Venue
- Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2024-10-27
- Pages
- 2087-2098
- OpenAlex
- W4403520284
- Cited by
- 16
- References
- 1
- Citations by year
- 2026: 11 · 2025: 3 · 2024: 2
- FWCI
- 4.72
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ant Group
- National Natural Science Foundation of China (62072046)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Signal Processing › Advanced Malware Detection Techniques
- Keywords
- Hugging Face, taint analysis, attack detection, reverse shell, credential theft, supply chain security
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
