Trusted AI in Multiagent Systems: An Overview of Privacy and Security for Distributed Learning
Open accessAbstract
Motivated by the advancing computational capacity of distributed end-user equipment (UE), as well as the increasing concerns about sharing private data, there has been considerable recent interest in machine learning (ML) and artificial intelligence (AI) that can be processed on distributed UEs. Specifically, in this paradigm, parts of an ML process are outsourced to multiple distributed UEs. Then, the processed information is aggregated on a certain level at a central server, which turns a centralized ML process into a distributed one and brings about significant benefits. However, this new distributed ML paradigm raises new risks in terms of privacy and security issues. In this article, we provide a survey of the emerging security and privacy risks of distributed ML from a unique perspective of information exchange levels, which are defined according to the key steps of an ML process, i.e., we consider the following levels: 1) the level of preprocessed data; 2) the level of learning models; 3) the level of extracted knowledge; and 4) the level of intermediate results. We explore and analyze the potential of threats for each information exchange level based on an overview of current state-of-the-art attack mechanisms and then discuss the possible defense methods against such threats. Finally, we complete the survey by providing an outlook on the challenges and possible directions for future research in this critical area.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Black-box models may still make poor predictions even in the absence of adversarial attacks due to significant biases and prejudices in the training set.
“Leading to significant biases and prejudices in the training set, black-box models may still make poor predictions even in the absence of adversarial attacks [28], [46], [47].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [46]
The ultimate level of transparency is retained by simulable models, next by decomposable structures, and subsequently by computationally transparent configurations.
“The ultimate level of transparency is retained by simulable models, next by decomposable structures, and subsequently by computationally transparent configurations [46], [50].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [46]
To assess the significance of an input characteristic, there are two work pathways: feature importance and activation maximization.
“To assess the significance of an input characteristic, there are two different work pathways which include feature importance and activation maximization [46], [53].”
From A Review of Trustworthy and Explainable Artificial Intelligence (XAI) (2023) · cites this paper as [46]
Cites
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772 · see citing works
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · see citing works
t-Closeness: Privacy Beyond k-Anonymity and l-Diversity
Li, N., Li, T., & Venkatasubramanian, S. (2007) · 2007 IEEE 23rd International Conference on Data Engineering · doi:10.1109/icde.2007.367856 · Cited by 3,451 · see citing works
Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias
Warner, S. L. (1965) · Journal of the American Statistical Association · doi:10.2307/2283137 · Cited by 3,263 · see citing works
Federated Learning With Differential Privacy: Algorithms and Performance Analysis
Wei, K., Li, J., Ding, M., Ma, C., Yang, H. H., Farokhi, F., Jin, S., Quek, T. Q. S., & Vincent Poor, H. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.2988575 · Cited by 2,400 · see citing works
Self-citationRAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response
Erlingsson, Ú., Pihur, V., & Korolova, A. (2014) · Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2660267.2660348 · Cited by 1,838 · see citing works
Privacy-preserving data publishing: A survey of recent developments
Fung, B. C. M., Wang, K., Chen, R., & Yu, P. S. (2010) · ACM Computing Surveys · doi:10.1145/1749603.1749605 · Cited by 1,704 · cited 1× · see citing works
“1) Difference between Security and Privacy: The concerns of security and privacy issues are different [256]–[258].”
BackgroundDeep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540 · cited 1× · see citing works
“, deep learning, have revolutionized the ways that information is extracted with ground-breaking successes in various areas [1].”
BackgroundHow to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495 · see citing works
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134012 · Cited by 1,377 · see citing works
Homer, N., Szelinger, S., Redman, M., Duggan, D., Tembe, W., Muehling, J., Pearson, J. V., Stephan, D. A., Nelson, S. F., & Craig, D. W. (2008) · PLoS Genetics · doi:10.1371/journal.pgen.1000167 · Cited by 1,221 · see citing works
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00057 · Cited by 861 · see citing works
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., & Pedarsani, R. (2019) · arXiv · doi:10.48550/arxiv.1909.13014 · Cited by 587 · see citing works
Stich, S. U., Cordonnier, J.-B., & Jaggi, M. (2018) · arXiv · doi:10.48550/arxiv.1809.07599 · Cited by 508 · see citing works
Jeong, E., Oh, S., Kim, H., Park, J., Bennis, M., & Kim, S.-L. (2018) · arXiv · doi:10.48550/arxiv.1811.11479 · Cited by 465 · see citing works
LOGAN: Membership Inference Attacks Against Generative Models
Hayes, J., Melis, L., Danezis, G., & De Cristofaro, E. (2019) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2019-0008 · Cited by 433 · see citing works
Communication-Efficient Federated Learning for Digital Twin Edge Networks in Industrial IoT
Lu, Y., Huang, X., Zhang, K., Maharjan, S., & Zhang, Y. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3010798 · Cited by 316 · see citing works
"You Might Also Like:" Privacy Risks of Collaborative Filtering
Calandrino, J. A., Kilzer, A., Narayanan, A., Felten, E. W., & Shmatikov, V. (2011) · 2011 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2011.40 · Cited by 311 · see citing works
Private predictive analysis on encrypted medical data
Bos, J. W., Lauter, K., & Naehrig, M. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.04.003 · Cited by 283 · cited 1× · see citing works
“Typical examples are, neural networks [78]–[80], matrix factorization [81], linear regressions [82], decision trees [83], and linear classifiers [84], [85].”
BackgroundWhen Federated Learning Meets Blockchain: A New Distributed Learning Paradigm
Ma, C., Li, J., Ding, M., Shi, L., Wang, T., Han, Z., & Poor, H. V. (2020) · arXiv · doi:10.48550/arxiv.2009.09338 · Cited by 225 · see citing works
Self-citationMore Than Privacy: Applying Differential Privacy in Key Areas of Artificial Intelligence
Zhu, T., Ye, D., Wang, W., Zhou, W., & Yu, P. S. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3014246 · Cited by 219 · see citing works
Strobel, V., Castelló Ferrer, E., & Dorigo, M. (2018) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/nlel1871 · Cited by 149 · see citing works
A Survey on Differentially Private Machine Learning [Review Article]
Gong, M., Xie, Y., Pan, K., Feng, K., & Qin, A. K. (2020) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2020.2976185 · Cited by 115 · see citing works
A Review of Privacy-Preserving Federated Learning for the Internet-of-Things
Briggs, C., Fan, Z., & Andras, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-70604-3_2 · Cited by 101 · see citing works
RobustFed: A Truth Inference Approach for Robust Federated Learning
Tahmasebian, F., Lou, J., & Xiong, L. (2022) · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · doi:10.1145/3511808.3557439 · Cited by 26 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Review of Trustworthy and Explainable Artificial Intelligence (XAI)
Chamola, V., Hassija, V., Sulthana, A. R., Ghosh, D., Dhingra, D., & Sikdar, B. (2023) · IEEE Access · doi:10.1109/access.2023.3294569 · Cited by 352 · see citing works
“The entire basis of tailor made advertisements for every person is one use of such distributed AI [21].”
BackgroundMcIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“Despite these efforts, challenges such as achieving true anonymity against correlation attacks highlight the complexities in effectively protecting against intrusive surveillance [240], [241].”
Background
- Authors
- Ma, C., Li, J., Wei, K., Liu, B., Ding, M., Yuan, L., Han, Z., & Vincent Poor, H.
- Affiliations
- Zhejiang Lab (CN)
- Nanjing University of Science and Technology (CN)
- Hong Kong Polytechnic University (HK)
- University of Technology Sydney (AU)
- Data61 (AU)
- Kyung Hee University (KR)
- Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ, USA
- Venue
- Proceedings of the IEEE
- Published
- 2023-09
- Volume
- 111
- Issue
- 9
- Pages
- 1097-1132
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W4386737174
- Cited by
- 96
- References
- 26
- Citations by year
- 2026: 24 · 2025: 32 · 2024: 30 · 2023: 9 · 2022: 1
- FWCI
- 8.41
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings of the IEEE (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Key Research and Development Program of China (2022YFF0712100)
- National Natural Science Foundation of China (62002170, 61902184)
- Fundamental Research Funds for the Central Universities (30921013104)
- Science and Technology on Information Systems Engineering Laboratory (WDZC20205250411)
- Future Network Grant of Provincial Education Board in Jiangsu
- Youth Foundation Project of Zhejiang Laboratory (K2023PD0AA01)
- Research Initiation Project of Zhejiang Laboratory
- U.S. National Science Foundation, U.S. Department of Transportation, Toyota and Amazon (CNS-2107216, CNS-2128368, CNS-2128448, CMMI-2222810, ECCS-2302469, ECCS-2335876)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Cryptography and Data Security
- Keywords
- trustworthy AI, multi-agent systems, distributed learning, privacy, security (computing), privacy attacks, defense mechanisms, data privacy, model privacy, intermediate outcomes, edge computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
