Distributed Machine Learning through Heterogeneous Edge Systems
Open accessAbstract
Many emerging AI applications request distributed machine learning (ML) among edge systems (e.g., IoT devices and PCs at the edge of the Internet), where data cannot be uploaded to a central venue for model training, due to their large volumes and/or security/privacy concerns. Edge devices are intrinsically heterogeneous in computing capacity, posing significant challenges to parameter synchronization for parallel training with the parameter server (PS) architecture. This paper proposes ADSP, a parameter synchronization scheme for distributed machine learning (ML) with heterogeneous edge systems. Eliminating the significant waiting time occurring with existing parameter synchronization models, the core idea of ADSP is to let faster edge devices continue training, while committing their model updates at strategically decided intervals. We design algorithms that decide time points for each worker to commit its model update, and ensure not only global model convergence but also faster convergence. Our testbed implementation and experiments show that ADSP outperforms existing parameter synchronization models significantly in terms of ML model convergence time, scalability and adaptability to large heterogeneity.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
[211] accelerates convergence by 62.4%.
“Convergence acceleration: 62.4%”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [211]
Cites
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,369 · see citing works
Federated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766 · see citing works
(2013) · Cited by 3,740 · see citing works
A Bridging Model for Parallel Computation
Valiant, L. G. (1990) · ACM eBooks · doi:10.1145/3749740.3749753 · Cited by 3,697 · see citing works
Some methods of speeding up the convergence of iteration methods
Polyak, B. T. (1964) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(64)90137-5 · Cited by 2,867 · see citing works
Adaptive Federated Learning in Resource Constrained Edge Computing Systems
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904348 · Cited by 2,232 · see citing works
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819 · see citing works
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134 · see citing works
The Impact of Virtualization on Network Performance of Amazon EC2 Data Center
Wang, G., & Ng, T. S. E. (2010) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2010.5461931 · Cited by 662 · see citing works
Project Adam: building an efficient and scalable deep learning training system
Chilimbi, T. M., Suzue, Y., Apacible, J., & Kalyanaraman, K. (2014) · Operating Systems Design and Implementation · doi:10.5555/2685048.2685094 · Cited by 614 · see citing works
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590 · see citing works
More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server
Ho, Q., Cipar, J., Cui, H., Kim, J. K., Lee, S., Gibbons, P. B., Gibson, G. A., Ganger, G. R., & Xing, E. P. (2013) · PubMed · doi:10.1184/r1/6475898 · Cited by 586 · see citing works
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDPreprint
Wang, J., & Joshi, G. (2018) · USENIX workshop on Tackling computer systems problems with machine learning techniques · doi:10.48550/arxiv.1810.08313 · Cited by 265 · see citing works
Heterogeneity-aware Distributed Parameter Servers
Jiang, J., Cui, B., Zhang, C., & Yu, L. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3035933 · Cited by 206 · see citing works
eSGD: Communication Efficient Distributed Deep Learning on the Edge
Tao, Z., & Li, Q. (2018) · Cited by 92 · see citing works
Wireless Network Intelligence at the Edge
Park, J., Samarakoon, S., Bennis, M., & Debbah, M. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2941458 · Cited by 92 · see citing works
Round-Robin Synchronization: Mitigating Communication Bottlenecks in Parameter Servers
Chen, C., Wang, W., & Li, B. (2019) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2019.8737587 · Cited by 90 · see citing works
Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUsPreprint
Hadjis, S., Zhang, C., Mitliagkas, I., & Ré, C. (2016) · arXiv.org · doi:10.48550/arxiv.1606.04487 · Cited by 68 · see citing works
Data-driven Task Allocation for Multi-task Transfer Learning on the Edge
Chen, Q., Zheng, Z., Hu, C., Wang, D., & Liu, F. (2019) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00107 · Cited by 48 · see citing works
Self-citationDistributed Machine Learning through Heterogeneous Edge Systems
Hu, H., Wang, D., & Wu, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6207 · Cited by 44 · see citing works
Self-citationStay Fresh: Speculative Synchronization for Fast Distributed Machine Learning
Zhang, C., Tian, H., Wang, W., & Yan, F. (2018) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2018.00020 · Cited by 25 · see citing works
Timed Dataflow: Reducing Communication Overhead for Distributed Machine Learning Systems
Sun, P., Wen, Y., Duong, T. N. B., & Yan, S. (2016) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads.2016.0146 · Cited by 16 · see citing works
Next Generation of DevOps: AIOps in Practice @Baidu
Qu, X., & Ha, J. (2017) · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 24 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498 · see citing works
“For more powerful edge devices, ADSP [211] lets them continue training while committing model aggregation at strategically decided intervals.”
BackgroundEdge deep learning in computer vision and medical diagnostics: a comprehensive survey
Xu, Y., Khan, T. M., Song, Y., & Meijering, E. H. W. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11033-5 · Cited by 174 · see citing works
Towards asynchronous federated learning for heterogeneous edge-powered internet of things
Chen, Z., Liao, W., Hua, K., Lu, C., & Yu, W. (2021) · Digital Communications and Networks · doi:10.1016/j.dcan.2021.04.001 · Cited by 126 · see citing works
Edge-Based Communication Optimization for Distributed Federated Learning
Wang, T., Liu, Y., Zheng, X., Dai, H., Jia, W., & Xie, M. (2021) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2021.3083263 · Cited by 120 · see citing works
A survey of mobility-aware Multi-access Edge Computing: Challenges, use cases and future directions
Singh, R., Sukapuram, R., & Chakraborty, S. (2022) · Ad Hoc Networks · doi:10.1016/j.adhoc.2022.103044 · Cited by 107 · see citing works
On-Device Learning Systems for Edge Intelligence: A Software and Hardware Synergy Perspective
Zhou, Q., Qu, Z., Guo, S., Luo, B., Guo, J., Xu, Z., & Akerkar, R. A. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3063147 · Cited by 50 · see citing works
L4L: Experience-Driven Computational Resource Control in Federated Learning
Zhan, Y., Li, P., Wu, L., & Guo, S. (2021) · IEEE Transactions on Computers · doi:10.1109/tc.2021.3068219 · Cited by 39 · see citing works
Distributed machine learning for energy trading in electric distribution system of the future
Wang, N., Li, J., Ho, S.-S., & Qiu, C. (2020) · The Electricity Journal · doi:10.1016/j.tej.2020.106883 · Cited by 29 · see citing works
Federated Optimization Under Intermittent Client Availability
Yan, Y., Niu, C., Ding, Y., Zheng, Z., Tang, S., Li, Q., Wu, F., Lyu, C., Feng, Y., & Chen, G. (2023) · INFORMS journal on computing · doi:10.1287/ijoc.2022.0057 · Cited by 28 · see citing works
Preemptive Scheduling for Distributed Machine Learning Jobs in Edge-Cloud Networks
Wang, N., Zhou, R., Jiao, L., Zhang, R., Li, B., & Li, Z. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3180772 · Cited by 23 · see citing works
Privacy preserving federated learning for full heterogeneity
Chen, K., Zhang, X., Zhou, X., Mi, B., Xiao, Y., Zhou, L., Wu, Z., Wu, L., & Wang, X. (2023) · ISA Transactions · doi:10.1016/j.isatra.2023.04.020 · Cited by 22 · see citing works
Blockchain-Empowered Federated Learning Through Model and Feature Calibration
Wang, Q., Liao, W., Guo, Y., McGuire, M. P., & Yu, W. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3311967 · Cited by 19 · see citing works
Job scheduling for distributed machine learning in optical WAN
Liu, L., Yu, H., Sun, G., Luo, L., Jin, Q., & Luo, S. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.06.007 · Cited by 19 · see citing works
Federated learning based on stratified sampling and regularization
Lu, C., Ma, W., Wang, R., Deng, S., & Wu, Y. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-022-00895-3 · Cited by 17 · see citing works
DPS: Dynamic Pricing and Scheduling for Distributed Machine Learning Jobs in Edge-Cloud Networks
Zhou, R., Wang, N., Huang, Y., Pang, J., & Chen, H. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3195765 · Cited by 15 · see citing works
Yoon, J., Byeon, Y.-H., Kim, J., & Lee, H. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3131407 · Cited by 17 · see citing works
EdgeMove: Pipelining Device-Edge Model Training for Mobile Intelligence
Dong, Z., He, Q., Chen, F., Jin, H., Gu, T., & Yang, Y. J. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583540 · Cited by 13 · see citing works
Liu, L., Zhou, P., Sun, G., Chen, X., Wu, T. T., Yu, H., & Guizani, M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127009 · Cited by 12 · see citing works
Shen, Z., Tang, Q., Zhou, T., Zhang, Y., Jia, Z., Yu, D., Zhang, Z., & Li, B. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3315847 · Cited by 11 · see citing works
Spread: Decentralized Model Aggregation for Scalable Federated Learning
Hu, C., Liang, H. H., Han, X. M., Liu, B. A., Cheng, D., & Wang, D. (2022) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3545008.3545030 · Cited by 11 · see citing works
Self-citationGWPF: Communication-efficient federated learning with Gradient-Wise Parameter Freezing
Yang, D., Gao, Y., Hu, B., Jin, A.-L., Wang, W., & You, Y. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110886 · Cited by 7 · see citing works
Quyoom, B., Wani, A. A., Lone, A. A., Dar, Z. A., Mir, B. A., & Peer, L. A. (2026) · Discover Plants. · doi:10.1007/s44372-026-00485-4 · Cited by 6 · see citing works
Asynchronous Federated Learning with Grey Wolf Optimization for the Heterogeneity IoT Devices
Kusumo, B. H., & Singh, P. (2024) · International Journal of Data Informatics and Intelligent Computing · doi:10.59461/ijdiic.v3i2.105 · Cited by 6 · see citing works
H-PS: A Heterogeneous-Aware Parameter Server With Distributed Neural Network Training
Xian, L., Li, B., Liu, J., Guo, Z., & Du, D. H.-C. (2021) · IEEE Access · doi:10.1109/access.2021.3060154 · Cited by 6 · see citing works
Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. EnvironmentsPreprint
Verbraeken, J., de Vos, M., & Pouwelse, J. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.11006 · Cited by 6 · see citing works
Cai, F., Yuan, D., Xie, M., He, W. P., Kong, L., Guo, W., Jiang, Y., & Cui, L. (2023) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws60048.2023.00045 · Cited by 4 · see citing works
PS+: A Simple yet Effective Framework for Fast Training on Parameter Server
Jin, A.-L., Xu, W., Guo, S., Hu, B., & Yeung, K. L. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3200518 · Cited by 4 · see citing works
Zhou, Y., Ye, Q., & Huang, H. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3614383 · Cited by 3 · see citing works
Container Image Similarity-Aware Resource Provisioning for Serverless Edge Computing
Zhou, A., Li, S., Ma, X., Zhang, Y., & Wang, S. (2023) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws60048.2023.00047 · Cited by 3 · see citing works
MIDDLE: A Mobility-Driven Device-Edge-Cloud Federated Learning Framework
Zhang, S., Zheng, Z., Wu, F., Li, B., Shao, Y., & Chen, G. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3543723 · Cited by 3 · see citing works
Accelerating distributed machine learning with model compression and graph partition
Duan, Y., & Wu, J. (2023) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2023.04.006 · Cited by 3 · see citing works
Online Scheduling Unbiased Distributed Learning over Wireless Edge Networks
Han, Z., Zhou, R., Pang, J., Cao, Y., & Tan, H. (2021) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads53394.2021.00080 · Cited by 3 · see citing works
SyncNet: Harmonizing Nodes for Efficient Learning
Roy, A., Gyanchandani, B. S., Sahu, J., & Rao, K. M. (2024) · International Conference on Software, Telecommunications and Computer Networks (SoftCOM) · doi:10.23919/softcom62040.2024.10721994 · Cited by 2 · see citing works
WBSP: Addressing stragglers in distributed machine learning with worker-busy synchronous parallel
Yang, D., Hu, B., Liu, A., Jin, A.-L., Yeung, K. L., & You, Y. (2024) · Parallel Computing · doi:10.1016/j.parco.2024.103092 · Cited by 2 · see citing works
FlexiNet: Lightweight Traffic Fingerprinting for Resource-Constrained Environments
Rouf, A., Taheri, R., & Charyyev, B. (2024) · Globecom · doi:10.1109/globecom52923.2024.10901059 · Cited by 2 · see citing works
ROG: A High Performance and Robust Distributed Training System for Robotic IoT
Guan, X., Sun, Z., Deng, S., Chen, X., Zhao, S., Zhang, Z., Duan, T., Wang, Y., Wu, C., Cui, Y., et al. (2022) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro56248.2022.00032 · Cited by 2 · see citing works
Self-citationModeling the Training Iteration Time for Heterogeneous Distributed Deep Learning Systems
Zeng, Y., Chen, B., Pan, P., Li, K., & Chen, G. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/2663115 · Cited by 2 · see citing works
Machine Learning Systems for Intelligent Services in the IoT: A SurveyPreprint
Toussaint, W., & Ding, A. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04950 · Cited by 2 · see citing works
Large language model augmentation technology for intelligent software development
Xiao, Z., Chen, H., Xiao, J. X., Yao, Z., & Wang, J. (2025) · International Conference on Electronic Information Engineering and Data Processing (EIEDP) · doi:10.1117/12.3066936 · Cited by 1 · see citing works
Reaction Latency Analysis of Message Synchronization in Edge-assisted Autonomous Driving
Jafarpourmarzouni, R., Sumaiya, F., Li, R., Guan, N., Wang, G., Zhou, P., & Dong, Z. (2025) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3736412 · Cited by 1 · see citing works
Artificial Intelligence Training at Edge
Wang, X., Han, Y., Leung, V. C. M., Niyato, D. T., Yan, X., & Chen, X. (2020) · Springer eBooks · doi:10.1007/978-981-15-6186-3_6 · Cited by 1 · see citing works
Dhariwal, N. (2023) · International Conference on Smart Technologies, Communication and Robotics (STCR) · doi:10.1109/stcr59085.2023.10396874 · Cited by 1 · see citing works
Research on Distributed Edge System Based on Artificial Intelligence Reasoning
Zheng, Z. (2023) · Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS) · doi:10.1109/acctcs58815.2023.00043
Uma Estratégia Hierárquica para Sistemas de Múltiplos Classificadores Distribuídos
de Salles, C. G., Brun, A. L., & Rodrigues, L. A. (2024) · doi:10.5753/wtf.2024.2517
Xu, X., Sun, J., Cui, G., Qi, L., Bilal, M., Dou, W., Cai, Z., & Crowcroft, J. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3615667
FedSM: A Federated Spectrum Management Architecture for 6G Network
Yan, J., He, Z., Hu, C., & Cheng, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-10466-3_18
Sultan, N., Boo, H., & Huh, E. (2025) · International Conference on Federated Learning Technologies and Applications (FLTA) · doi:10.1109/flta67013.2025.11336468
- Authors
- Hu, H., Wang, D., & Wu, C.
- Affiliations
- University of Hong Kong (HK)
- Hong Kong Polytechnic University (HK)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Volume
- 34
- Issue
- 05
- Pages
- 7179-7186
- ISSN
- 2159-5399 (ISSN-L)
- arXiv
- 1911.06949
- OpenAlex
- W2998174899
- Cited by
- 47
- References
- 24
- Citations by year
- 2026: 1 · 2025: 8 · 2024: 6 · 2023: 11 · 2022: 7 · 2021: 8 · 2020: 6
- FWCI
- 6.71
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Hong Kong Polytechnic University (1-ZVPZ)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- distributed learning, edge computing, parameter synchronization, model convergence, IoT devices, privacy-preserving training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
