The DOI matches its Crossref record; 62 of the 64 references it cites were found in a registry or an index.
Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning
Open accessAbstract
While recent works have indicated that federated learning (FL) may be vulnerable to poisoning attacks by compromised clients, their real impact on production FL systems is not fully understood. In this work, we aim to develop a comprehensive systemization for poisoning attacks on FL by enumerating all possible threat models, variations of poisoning, and adversary capabilities. We specifically put our focus on un-targeted poisoning attacks, as we argue that they are significantly relevant to production FL deployments. We present a critical analysis of untargeted poisoning attacks under practical, production FL environments by carefully characterizing the set of realistic threat models and adversarial capabilities. Our findings are rather surprising: contrary to the established belief, we show that FL is highly robust in practice even when using simple, low-cost defenses. We go even further and propose novel, state-of-the-art data and model poisoning attacks, and show via an extensive set of experiments across three benchmark datasets how (in)effective poisoning attacks are in the presence of simple defense mechanisms. We aim to correct previous misconceptions and offer concrete guidelines to conduct more accurate (and more realistic) research on this topic.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Virat et al. [23] argued that there are two kinds of label-flipping strategies, namely static label flipping (SLF) and dynamic label flipping (DLF).”
From Poisoning attacks in federated learning: A survey (2023) · cites this paper as [23]
In the citing paper
“To generate a malicious model w k+1 0 , the attacker can use a specific dataset (e.g., different kinds of green cars labeled as birds [29]) or train the model in a benign dataset by projected gradient ascent (while the benign clients use stochastic gradient descent) algorithm [23], [67], [77].”
From Poisoning attacks in federated learning: A survey (2023) · cites this paper as [23]
In the citing paper
“To avoid detection, Virat et al. [23] also state that model updates should not deviate too far from benign updates to avoid detection.”
From Poisoning attacks in federated learning: A survey (2023) · cites this paper as [23]
In the citing paper
“However, model poisoning attacks tend to work better in practice because they can directly modify local model parameters [11], [23], [33], [40].”
From Poisoning attacks in federated learning: A survey (2023) · cites this paper as [23]
In the citing paper
“This attack can be applied on a batch of target data samples all at the same time [73].”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [73]
In the citing paper
“One such attack is the gradient ascent attack [73], where the attacker runs gradient ascent on a target data sample and observes whether its increased loss can be drastically reduced in the next communication round, if so, the sample is very likely to be in the training set.”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [73]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Self-citationAdvances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Self-citationFederated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766
EMNIST: Extending MNIST to handwritten letters
Cohen, G., Afshar, S., Tapson, J. C., & André, V. S. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966217 · Cited by 2,410
Towards Federated Learning at Scale: System Design
Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečný, J., Mazzocchi, S., McMahan, H. B., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.01046 · Cited by 2,334
Self-citationNeural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H.-T., & Zhao, B. Y. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00031 · Cited by 1,577
Wild patterns: Ten years after the rise of adversarial machine learning
Biggio, B., & Roli, F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.023 · Cited by 1,563
How to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495
(2017) · neural information processing systems · Cited by 1,413
Poisoning Attacks against Support Vector Machines
Biggio, B., Nelson, B., & Laskov, P. (2012) · arXiv · doi:10.48550/arxiv.1206.6389 · Cited by 1,331
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194
Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
Yin, D., Chen, Y., Ramchandran, K., & Bartlett, P. (2018) · arXiv · doi:10.48550/arxiv.1803.01498 · Cited by 1,130
Robust Aggregation for Federated Learning
Pillutla, K., Kakade, S. M., & Harchaoui, Z. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3153135 · Cited by 1,068
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
Chen, X., Liu, C., Li, B., Lu, K., & Song, D. (2017) · arXiv · doi:10.48550/arxiv.1712.05526 · Cited by 1,039
Data Poisoning Attacks Against Federated Learning Systems
Tolpegin, V., Truex, S., Gursoy, M. E., & Liu, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58951-6_24 · Cited by 927
LEAF: A Benchmark for Federated Settings
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., & Talwalkar, A. (2018) · arXiv · doi:10.48550/arxiv.1812.01097 · Cited by 894
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00057 · Cited by 861
Learning Differentially Private Recurrent Language Models
McMahan, H. B., Ramage, D., Talwar, K., & Zhang, L. (2017) · arXiv · doi:10.48550/arxiv.1710.06963 · Cited by 849
Self-citationAdaptive Federated Optimization
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., & McMahan, H. B. (2020) · arXiv · doi:10.48550/arxiv.2003.00295 · Cited by 840
The security of machine learning
Barreno, M., Nelson, B. A., Joseph, A. D., & Tygar, J. D. (2010) · Machine Learning · doi:10.1007/s10994-010-5188-5 · Cited by 839
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning
Fang, M., Cao, X., Jia, J., & Gong, N. Z. (2019) · arXiv · doi:10.48550/arxiv.1911.11815 · Cited by 828
Estimating a Dirichlet Distribution
Minka, T. P. (2000) · Cited by 778
Ensemble Distillation for Robust Model Fusion in Federated LearningPreprint
Lin, T., Kong, L., Stich, S. U., & Jaggi, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07242 · Cited by 761
Analyzing Federated Learning through an Adversarial Lens
Bhagoji, A. N., Chakraborty, S., Mittal, P., & Calo, S. (2018) · arXiv · doi:10.48550/arxiv.1811.12470 · Cited by 703
The Hidden Vulnerability of Distributed Learning in Byzantium
Mhamdi, E. M. E., Guerraoui, R., & Rouault, S. (2018) · arXiv · doi:10.48550/arxiv.1802.07927 · Cited by 648
Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural NetworksPreprint
Shafahi, A., Huang, W., Najibi, M., Suciu, O., Studer, C., Dumitraş, T., & Goldstein, T. (2018) · arXiv (Cornell University) · doi:10.3929/ethz-b-000460348 · Cited by 626
Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning
Shejwalkar, V., & Houmansadr, A. (2021) · Proceedings 2021 Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24498 · Cited by 615
Self-citationTowards Poisoning of Deep Learning Algorithms with Back-gradient Optimization
Muñoz-González, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V., Lupu, E. C., & Roli, F. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140451 · Cited by 549
Ditto: Fair and Robust Federated Learning Through PersonalizationPreprint
Li, T., Hu, S., Beirami, A., & Smith, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.04221 · Cited by 536
Federated Learning: Collaborative Machine Learning without Centralized Training Data
Abhishek, V. A., Binny, S., Johan, T. R., Raj, N., & Thomas, V. (2022) · International Journal of Engineering Technology and Management Sciences · doi:10.46647/ijetms.2022.v06i05.052 · Cited by 489
DBA: Distributed Backdoor Attacks against Federated Learning
Xie, C., Huang, K., Chen, P., & Li, B. (2020) · International Conference on Learning Representations · Cited by 472
Attack of the tails: Yes, you really can backdoor federated learning
Wang, H., Sreenivasan, K., Rajput, S., Vishwakarma, H., Agarwal, S., Sohn, J.-Y., Lee, K., & Papailiopoulos, D. (2020) · arXiv · doi:10.48550/arxiv.2007.05084 · Cited by 406
Li, L., Xu, W., Chen, T., Giannakis, G. B., & Ling, Q. H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011544 · Cited by 392
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., & Goldstein, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3162397 · Cited by 362
Can You Really Backdoor Federated Learning?
Sun, Z., Kairouz, P., Suresh, A. T., & McMahan, H. B. (2019) · arXiv · doi:10.48550/arxiv.1911.07963 · Cited by 348
Self-citation(2019) · neural information processing systems · Cited by 318
Data Poisoning Attacks on Federated Machine Learning
Sun, G., Cong, Y., Dong, J., Wang, Q., Lyu, L., & Liu, J. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3128646 · Cited by 310
The Limitations of Federated Learning in Sybil Settings
Fung, C., Yoon, C. J. M., & Beschastnikh, I. (2020) · Recent Advances in Intrusion Detection · Cited by 265
signSGD with Majority Vote is Communication Efficient And Fault TolerantPreprint
Bernstein, J., Zhao, J., Azizzadenesheli, K., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05291 · Cited by 234
Support vector machines under adversarial label contamination
Huang, X. X., Biggio, B., Nelson, B. A., Xiao, H., Eckert, C. M., & Roli, F. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.08.081 · Cited by 220
Salvaging Federated Learning by Local AdaptationPreprint
Yu, T., Bagdasaryan, E., & Shmatikov, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04758 · Cited by 220
Byzantine Stochastic Gradient DescentPreprint
Alistarh, D., Allen-Zhu, Z., & Li, J. Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08917 · Cited by 217
A Taxonomy of Attacks on Federated Learning
Jere, M. S., Farnan, T., & Koushanfar, F. (2021) · IEEE Security & Privacy · doi:10.1109/msec.2020.3039941 · Cited by 217
Generative Poisoning Attack Method Against Neural NetworksPreprint
Yang, C., Wu, Q., Li, H. H., & Chen, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01340 · Cited by 215
Adversarial Label Flips Attack on Support Vector Machines
Xiao, H., Huang, X. X., & Claudia, E. (2012) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-61499-098-7-870 · Cited by 208
DRACO: Byzantine-resilient Distributed Training via Redundant GradientsPreprint
Chen, L., Wang, H., Charles, Z., & Papailiopoulos, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09877 · Cited by 191
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge TransferPreprint
Chang, H., Shejwalkar, V., Shokri, R., & Houmansadr, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11279 · Cited by 176
Self-citationGeneralized Byzantine-tolerant SGDPreprint
Xie, C., Koyejo, O. O., & Gupta, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10116 · Cited by 172
Provably Secure Federated Learning against Malicious Clients
Cao, X., Jia, J., & Gong, N. Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16849 · Cited by 141
CRFL: Certifiably Robust Federated Learning against Backdoor AttacksPreprint
Xie, C., Chen, M., Chen, P., & Li, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08283 · Cited by 121
Mitigating Backdoor Attacks in Federated LearningPreprint
Wu, C., Xian, Y., Zhu, S., & Mitra, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01767 · Cited by 120
IBM Federated Learning: an Enterprise Framework White Paper V0.1
Ludwig, H., Baracaldo, N., Thomas, G., Zhou, Y., Anwar, A., Rajamoni, S., Ong, Y., Radhakrishnan, J., Verma, A., Sinn, M., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.10987 · Cited by 106
Attack-Resistant Federated Learning with Residual-based ReweightingPreprint
Fu, S., Xie, C., Li, B., & Chen, Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11464 · Cited by 91
Poisoning Attacks with Generative Adversarial NetsPreprint
Muñoz-González, L., Pfitzner, B., Russo, M., Carnerero-Cano, J., & Lupu, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07773 · Cited by 79
On the Practicality of Integrity Attacks on Document-Level Sentiment Analysis
Newell, A., Potharaju, R., Xiang, L., & Nita-Rotaru, C. (2014) · doi:10.1145/2666652.2666661 · Cited by 49
Federated Evaluation and Tuning for On-Device Personalization: System Design & ApplicationsPreprint
Paulik, M., Seigel, M., Mason, H., Telaar, D., Kluivers, J., van Dalen, R., Lau, C. W., Carlson, L., Granqvist, F., Vandevelde, C., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08503 · Cited by 37
Byzantine-Resilient SGD in High Dimensions on Heterogeneous Data
Data, D., & Diggavi, S. N. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9518248 · Cited by 30
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and DefensesPreprint
Goldblum, M., Tsipras, D., Xie, C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.10544 · Cited by 24
SGD: Decentralized Byzantine Resilience.Preprint
Mhamdi, E. M. E., Guerraoui, R., Guirguis, A., & Rouault, S. (2019) · arXiv (Cornell University) · Cited by 13
Research works printed but not linked: 1
- 2020, Google Workshop on Federated Learning and Analytics
Cited by
Sorted by citations
Privacy and Robustness in Federated Learning: Attacks and Defenses
Lyu, L., Yu, H., Ma, X., Chen, C., Sun, L., Zhao, J., Yang, Q., & Yu, P. S. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3216981 · Cited by 548
Poisoning attacks in federated learning: A survey
Xia, G., Chen, J., Yu, C., & Ma, J. (2023) · IEEE Access · doi:10.1109/access.2023.3238823 · Cited by 212
MPAF: Model Poisoning Attacks to Federated Learning based on Fake Clients
Cao, X., & Gong, N. Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00383 · Cited by 205
The Impact of Adversarial Attacks on Federated Learning: A Survey
Kumar, K. N., Mohan, C. K., & Cenkeramaddi, L. R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3322785 · Cited by 209
Adversarial machine learning :
Vassilev, A., Oprea, A., Fordyce, A. J., & Anderson, H. S. (2024) · doi:10.6028/nist.ai.100-2e2023 · Cited by 230
Metadata
- Authors
- Shejwalkar, V., Houmansadr, A., Kairouz, P., & Ramage, D.
- Affiliations
- University of Massachusetts Amherst (US)
- Google (United States) (US)
- Venue
- 2022 IEEE Symposium on Security and Privacy (SP)
- Publisher
- IEEE
- Published
- 2022-05
- Pages
- 1354-1371
- ISSN
- 1081-6011 (ISSN-L)
- OpenAlex
- W4288057793
- Cited by
- 292
- References
- 64
- Citations by year
- 2026: 53 · 2025: 78 · 2024: 97 · 2023: 52 · 2022: 12
- FWCI
- 32.95
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1553301, 1953786, 1739462)
- U.S. Department of Defense
- Defense Advanced Research Projects Agency
- Naval Information Warfare Center Pacific
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- federated learning, poisoning attacks, untargeted attacks, threat model, adversary capabilities, data poisoning, model poisoning, defense mechanisms, production systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
