Communication-Efficient Edge AI: Algorithms and Systems
Open accessThe DOI matches its Crossref record; 241 of the 242 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Artificial intelligence (AI) has achieved remarkable breakthroughs in a wide range of fields, ranging from speech processing, image classification to drug discovery. This is driven by the explosive growth of data, advances in machine learning (especially deep learning), and the easy access to powerful computing resources. Particularly, the wide scale deployment of edge devices (e.g., IoT devices) generates an unprecedented scale of data, which provides the opportunity to derive accurate models and develop various intelligent applications at the network edge. However, such enormous data cannot all be sent to the cloud for processing, due to the varying channel quality, traffic congestion and/or privacy concerns, and the enormous energy consumption. By pushing inference and training processes of AI models to edge nodes, edge AI has emerged as a promising alternative. AI at the edge requires close cooperation amongedge devices, such as smart phones and smart vehicles, andedge serversat the wireless access points and base stations, which however result in heavy communication overheads. In this paper, we present a comprehensive survey of the recent developments in various techniques for overcoming these communication challenges. Specifically, we first identify key communication challenges in edge AI systems. We then introduce communication-efficient techniques, from both algorithmic and system perspectives for training and inference tasks at the network edge. Potential future research directions are also highlighted.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“With the latest innovations in Artificial Intelligence (AI) efficiency, the proliferation of IoT devices, and the rise of edge computing, the potential of edge AI has now been unlocked [19].”
From Edge AI: A survey (2023) · cites this paper as [19]
In the citing paper
“Edge AI provides a promising solution for connected intelligence by enabling data collection, processing, transmission, and consumption at the network edge [29], [30].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [29]
In the citing paper
“Edge device distributed inference based on wireless MapReduce enjoys the advantages of providing low-latency, high-accurate, scalable, and resilient services for edge devices without accessing the cloud data center [12], [29].”
From Edge artificial intelligence for 6g: vision, enabling technologies, and applications (2022) · cites this paper as [29]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“…of computing resources, artificial intelligence (AI) has achieved breakthroughs in a wide range of applications, including speech processing [1], image classi-fication [2] and reinforcement learning [3], etc. AI is expected to affect significant segments of many vertical industries and our…”
BackgroundOn the limited memory BFGS method for large scale optimization
Liu, D. C., & Nocedal, J. (1989) · Mathematical Programming · doi:10.1007/bf01589116 · Cited by 8,560
Vector Quantization and Signal Compression
Gersho, A., & Gray, R. M. (1992) · Springer eBooks · doi:10.1007/978-1-4615-3626-0 · Cited by 7,003
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210 · cited 1×
“To unleash its full potential, the upcoming edge AI [9], [17] shall rely on advances in various aspects, including the smart design of distributed learning algorithms and system architectures, supported by efficient communication protocols.”
BackgroundA Survey on Mobile Edge Computing: The Communication Perspective
Mao, Y., You, C., Zhang, J., Huang, K., & Letaief, K. B. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2745201 · Cited by 5,607 · cited 7×
“…training an AI model (e.g., deep neural networks) generally requires intensive computing resources, the rapid development of mobile edge computing can provide cloud-computing capabilities at the edge of the mobile network [12], [13], and make the application of AI to edges much more efficient.”
BackgroundInfluentialSelf-citationDeep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733 · cited 1×
“…has achieved breakthroughs in a wide range of applications, including speech processing [1], image classi-fication [2] and reinforcement learning [3], etc. AI is expected to affect significant segments of many vertical industries and our daily life, such as intelligent vehicles [4] and tactile…”
BackgroundA Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408
Towards Smart and Reconfigurable Environment: Intelligent Reflecting Surface Aided Wireless Network
Wu, Q., & Zhang, R. (2020) · IEEE Communications Magazine · doi:10.1109/mcom.001.1900107 · Cited by 4,308
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 2×
“Such extremely high network bandwidth and low latency requirements would place unprecedented pressures on traditional cloud-based AI, where massive sensors/embedded devices transfer collected data to the cloud [8], often under varying network qualities (e.”
BackgroundFederated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
Federated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766
(1989) · Cited by 3,214
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455 · cited 6×
“To unleash its full potential, the upcoming edge AI [9], [18] shall rely on advances in various aspects, including the smart design of distributed learning algorithms and system architectures, supported by efficient communication protocols.”
BackgroundInfluentialSelf-citationBoyd, S., Ghosh, A., Prabhakar, B., & Shah, D. (2006) · IEEE Transactions on Information Theory · doi:10.1109/tit.2006.874516 · Cited by 2,439
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 1×
“For instance, it was shown in [162] that training with large minibatch sizes up to 8192 images achieves the same accuracy as small mini-batch settings by adjusting learning rates as a function of mini-batch size.”
BackgroundAdaptive Federated Learning in Resource Constrained Edge Computing Systems
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2904348 · Cited by 2,232
The Roadmap to 6G: AI Empowered Wireless Networks
Letaief, K. B., Chen, W., Shi, Y., Zhang, J., & Zhang, Y.-J. A. (2019) · IEEE Communications Magazine · doi:10.1109/mcom.2019.1900271 · Cited by 2,207
Self-citationBinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180
Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1602.02830 · Cited by 2,106
Introduction to Derivative-Free Optimization
Conn, A. R., Scheinberg, K., & Vicente, L. N. (2009) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9780898718768 · Cited by 1,840
Multi-Cell MIMO Cooperative Networks: A New Look at Interference
Gesbert, D., Hanly, S., Huang, H., Shamai Shitz, S., Simeone, O., & Yu, W. (2010) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2010.101202 · Cited by 1,760
Chen, P.-Y., Zhang, H., Sharma, Y., Yi, J., & Hsieh, C.-J. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140448 · Cited by 1,656
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · International Conference on Machine Learning · Cited by 1,590
A Joint Learning and Communications Framework for Federated Learning Over Wireless Networks
Chen, M., Yang, Z., Saad, W., Yin, C., Poor, H. V., & Cui, S. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3024629 · Cited by 1,547
A Joint Learning and Communications Framework for Federated Learning over Wireless Networks
Chen, M., Yang, Z., Saad, W., Yin, C., Poor, H. V., & Cui, S. (2019) · arXiv · doi:10.48550/arxiv.1909.07972 · Cited by 1,547
An overview of limited feedback in wireless communication systems
Love, D., Heath, R., N. Lau, V., Gesbert, D., Rao, B., & Andrews, M. (2008) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2008.081002 · Cited by 1,503
(1993) · CaltechAUTHORS (California Institute of Technology) · Cited by 1,498
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1404.0736 · Cited by 1,414
Deep Learning with Limited Numerical Precision
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015) · arXiv · doi:10.48550/arxiv.1502.02551 · Cited by 1,364 · cited 1×
“At the training stage, it has been shown that DNNs can be trained using only 16-bit wide fixed-point number representation by stochastic rounding [185], which induces little to no degradation in the classification accuracy.”
MethodBranchyNet: Fast inference via early exiting from deep neural networks
Teerapittayanon, S., McDanel, B., & Kung, H. T. (2016) · 2016 23rd International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.2016.7900006 · Cited by 1,326
(2019) · SGEM International Multidisciplinary Scientific Conferences on SOCIAL SCIENCES and ARTS Proceedings · doi:10.5593/sgemsocial2019v/1.1 · Cited by 1,298
Map-Reduce for Machine Learning on Multicore
Chu, C.-T., Kim, S. K., Lin, Y.-A., Yu, Y., Bradski, G., Ng, A. Y., & Olukotun, K. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0040 · Cited by 1,229
Quantized Convolutional Neural Networks for Mobile Devices
Wu, J., Leng, C., Wang, Y., Hu, Q., & Cheng, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.521 · Cited by 1,212
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180
Speeding up Convolutional Neural Networks with Low Rank Expansions
Jaderberg, M., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.88 · Cited by 1,164
Energy Efficient Federated Learning Over Wireless Communication Networks
Yang, Z., Chen, M., Saad, W., Hong, C. S., & Shikh-Bahaei, M. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3037554 · Cited by 1,153
Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence
Deng, S., Zhao, H., Fang, W., Yin, J., Dustdar, S., & Zomaya, A. Y. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2984887 · Cited by 1,147
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134
Parallelized Stochastic Gradient Descent
Zinkevich, M., Weimer, M., Li, L., & Smola, A. J. (2010) · Cited by 1,124
Federated Learning via Over-the-Air Computation
Yang, K., Jiang, T., Shi, Y., & Ding, Z. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2961673 · Cited by 1,084 · cited 5×
“The connection between data aggregation from distributed nodes in edge training and the in-network computation problem [20] in wireless sensor networks has been established in [21], which proposed an over-the-air computation approach for fast model”
BackgroundMethodInfluentialSelf-citationClient-Edge-Cloud Hierarchical Federated Learning
Liu, L., Zhang, J., Song, S. H., & Letaief, K. B. (2020) · ICC 2020 - 2020 IEEE International Conference on Communications (ICC) · doi:10.1109/icc40277.2020.9148862 · Cited by 1,033
Self-citationCompressing Neural Networks with the Hashing Trick
Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y. (2015) · arXiv · doi:10.48550/arxiv.1504.04788 · Cited by 974
Kang, J., Xiong, Z., Niyato, D., Xie, S., & Zhang, J. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2940820 · Cited by 953
Self-citationEdge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing
Li, E., Zeng, L., Zhou, Z., & Chen, X. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2946140 · Cited by 929
Using recurrent neural network models for early detection of heart failure onset
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw112 · Cited by 928
Compressing Deep Convolutional Networks using Vector Quantization
Gong, Y., Liu, L., Yang, M., & Bourdev, L. D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6115 · Cited by 925
Privacy-preserving distributed mining of association rules on horizontally partitioned data
Clifton, C., Kantarcioglu, M., Clifton, C., & Member, S. (2010) · Cited by 908
QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
Alistarh, D., Grubic, D., Li, J., Tomioka, R., & Vojnović, M. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.48550/arxiv.1610.02132 · Cited by 859
Predicting Parameters in Deep Learning
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N. (2013) · arXiv · doi:10.48550/arxiv.1306.0543 · Cited by 853
Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices
Teerapittayanon, S., McDanel, B., & Kung, H. T. (2017) · 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2017.226 · Cited by 843
Speeding Up Distributed Machine Learning Using Codes
Lee, K., Lam, M., Pedarsani, R., Papailiopoulos, D., & Ramchandran, K. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2736066 · Cited by 829
Random Gradient-Free Minimization of Convex Functions
Nesterov, Y., & Spokoiny, V. (2017) · Foundations of Computational Mathematics · doi:10.1007/s10208-015-9296-2 · Cited by 822
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., & Zaharia, M. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 818
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765695 · Cited by 817 · cited 1×
“Therefore, model compression approaches [27], [28] have attracted much attention for deploying DNN models at network edges.”
MethodNeurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge
Kang, Y., Hauswald, J., Gao, C., Rovinski, A., Mudge, T., Mars, J., & Tang, L. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3093337.3037698 · Cited by 806
Broadband Analog Aggregation for Low-Latency Federated Edge Learning
Zhu, G., Wang, Y., & Huang, K. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2946245 · Cited by 761
Computation Over Multiple-Access Channels
Nazer, B., & Gastpar, M. (2007) · IEEE Transactions on Information Theory · doi:10.1109/tit.2007.904785 · Cited by 753
Distributed learning of deep neural network over multiple agents
Gupta, O., & Raskar, R. (2018) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2018.05.003 · Cited by 726
Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge
Kang, Y., Hauswald, J., Gao, C., Rovinski, A., Mudge, T., Mars, J., & Tang, L. (2017) · Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3037697.3037698 · Cited by 673
SecureBoost: A Lossless Federated Learning Framework
Cheng, K., Fan, T., Jin, Y., Liu, Y., Chen, T., Papadopoulos, D., & Yang, Q. (2021) · IEEE Intelligent Systems · doi:10.1109/mis.2021.3082561 · Cited by 669
SecureBoost: A Lossless Federated Learning Framework
Cheng, K., Fan, T., Jin, Y., Liu, Y., Chen, T., Papadopoulos, D., & Yang, Q. (2019) · arXiv · doi:10.48550/arxiv.1901.08755 · Cited by 668
Artificial intelligence-enabled healthcare delivery
Reddy, S., Fox, J., & Purohit, M. P. (2019) · Journal of the Royal Society of Medicine · doi:10.1177/0141076818815510 · Cited by 635
Privacy-preserving k -means clustering over vertically partitioned data
Vaidya, J., & Clifton, C. (2003) · Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/956750.956776 · Cited by 626 · cited 2×
“Vaidya and Clifton [118] proposed a privacy-preserving k - means algorithm in the vertical architecture with secure multi-party computation.”
BackgroundMethodFederated Learning Over Wireless Fading Channels
Amiri, M. M., & Gunduz, D. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.2974748 · Cited by 625 · cited 2×
“• Fast aggregation via over-the-air computation [21], [84], [85], [86] • Aggregation frequency control with limited bandwidth and computation resources [87], [88], [89] • Data reshuffling via index coding and pliable index coding for improving training performance [90], [91], [92] • Straggler…”
BackgroundA Tour of Reinforcement Learning: The View from Continuous Control
Recht, B. (2019) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-053018-023825 · Cited by 609
Toward an Intelligent Edge: Wireless Communication Meets Machine Learning
Zhu, G., Liu, D., Du, Y., You, C., Zhang, J., & Huang, K. (2020) · IEEE Communications Magazine · doi:10.1109/mcom.001.1900103 · Cited by 606
Self-citationLow-rank matrix factorization for Deep Neural Network training with high-dimensional output targets
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638949 · Cited by 593 · cited 2×
“…sketching [62], [63], [64], network pruning [65], [66], [67], [27], [68], [69], [70], [71], sparse regularization [72], [73], [74], and structural matrix designing for minimizing communication bandwidth [75], [76], [77], [78], [79], [80], [81], [82], [83] Communication-EfficientEdgeAISystems”
BackgroundWhen Mobile Blockchain Meets Edge Computing
Xiong, Z., Zhang, Y., Niyato, D., Wang, P., & Han, Z. (2018) · IEEE Communications Magazine · doi:10.1109/mcom.2018.1701095 · Cited by 551
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
Mohammadi Amiri, M., & Gunduz, D. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.2981904 · Cited by 548
Federated Optimization:Distributed Optimization Beyond the Datacenter
Konečný, J., McMahan, B., & Ramage, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.03575 · Cited by 544
signSGD: Compressed Optimisation for Non-Convex Problems
Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., & Anandkumar, A. (2018) · arXiv · doi:10.48550/arxiv.1802.04434 · Cited by 541
Mobile Edge Intelligence and Computing for the Internet of Vehicles
Zhang, J., & Letaief, K. B. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2947490 · Cited by 537
Self-citationNetwork Topology and Communication-Computation Tradeoffs in Decentralized Optimization
Nedic, A., Olshevsky, A., & Rabbat, M. G. (2018) · Proceedings of the IEEE · doi:10.1109/jproc.2018.2817461 · Cited by 534
The Convergence of Sparsified Gradient MethodsPreprint
Alistarh, D., Hoefler, T., Johansson, M., Khirirat, S., Konstantinov, N., & Renggli, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10505 · Cited by 525
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04836 · Cited by 525
Scalable distributed DNN training using commodity GPU cloud computing
Strom, N. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-354 · Cited by 521
DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters
Zhao, Z., Barijough, K. M., & Gerstlauer, A. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2858384 · Cited by 502
Horovod: fast and easy distributed deep learning in TensorFlow
Sergeev, A., & Balso, M. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05799 · Cited by 472
Predicting Parameters in Deep Learning
Shakibi, B. (2014) · University of British Columbia · doi:10.14288/1.0165555 · Cited by 462
A Stochastic Quasi-Newton Method for Large-Scale Optimization
Byrd, R. H., Hansen, S. L., Nocedal, J., & Singer, Y. (2016) · SIAM Journal on Optimization · doi:10.1137/140954362 · Cited by 459
A Fundamental Tradeoff Between Computation and Communication in Distributed Computing
Li, S., Maddah-Ali, M. A., Yu, Q., & Avestimehr, A. S. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2756959 · Cited by 451
Seide, F., Fu, H., Droppo, J., Li, G., & Yu, D. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-274 · Cited by 443
Hardy, S., Henecka, W., Ivey-Law, H., Nock, R., Patrini, G., Smith, G., & Thorne, B. (2017) · arXiv · doi:10.48550/arxiv.1711.10677 · Cited by 431
Gradient Coding: Avoiding Stragglers in Distributed Learning
Tandon, R., Lei, Q., Dimakis, A. G., & Karampatziakis, N. (2017) · International Conference on Machine Learning · Cited by 431
A Stochastic Quasi-Newton Method for Online Convex Optimization
Schraudolph, N. N., Yu, J., & Günter, S. (2007) · ANU Open Research (Australian National University) · Cited by 430
Communication-Efficient Distributed Optimization using an Approximate Newton-type Method
Shamir, O., Srebro, N., & Zhang, T. (2013) · arXiv · doi:10.48550/arxiv.1312.7853 · Cited by 424 · cited 3×
“(c) Second-order method: DANE [54].”
MethodInfluentialHierarchical Federated Learning ACROSS Heterogeneous Cellular Networks
Abad, M. S. H., Ozfatura, E., GUndUz, D., & Ercetin, O. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054634 · Cited by 424
Bandwidth optimal all-reduce algorithms for clusters of workstations
Patarasuk, P., & Yuan, X. (2009) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2008.09.002 · Cited by 409
Reconfigurable-Intelligent-Surface Empowered Wireless Communications: Challenges and Opportunities
Yuan, X., Zhang, Y.-J. A., Shi, Y., Yan, W., & Liu, H. (2021) · IEEE Wireless Communications · doi:10.1109/mwc.001.2000256 · Cited by 407
Self-citationLearning Structured Sparsity in Deep Neural Networks
Wen, W. X., Wu, C., Wang, Y., Chen, Y., & Li, H. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03665 · Cited by 402
Machine Learning at the Network Edge: A Survey
Murshed, M. G. S., Murphy, C., Hou, D., Khan, N., Ananthanarayanan, G., & Hussain, F. (2022) · ACM Computing Surveys · doi:10.1145/3469029 · Cited by 402
Guidelines for reinforcement learning in healthcare
Gottesman, O., Johansson, F., Komorowski, M., Faisal, A., Sontag, D., Doshi-Velez, F., & Celi, L. A. (2019) · Nature Medicine · doi:10.1038/s41591-018-0310-5 · Cited by 395
Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
Chen, Y., Su, L., & Xu, J. (2017) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3154503 · Cited by 390
Petuum: A New Platform for Distributed Machine Learning on Big Data
Xing, E. P., Ho, Q., Dai, W., Kim, J. K., Wei, J., Lee, S., Zheng, X., Xie, P., Kumar, A., & Yu, Y. (2015) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2015.2472014 · Cited by 388
Zhu, G., Du, Y., Gunduz, D., & Huang, K. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2020.3039309 · Cited by 376 · cited 1×
“It should be noted that digital modulation schemes for over-the-air computation are advocated in [191], [192], [193], [194] due to its easier implementation on the existing communication systems and its less stringent requirement of synchronization compared”
BackgroundFast ConvNets Using Group-Wise Brain Damage
Lebedev, V., & Lempitsky, V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.280 · Cited by 373
Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations
Duchi, J. C., Jordan, M. I., Wainwright, M. J., & Wibisono, A. (2015) · IEEE Transactions on Information Theory · doi:10.1109/tit.2015.2409256 · Cited by 370
Glimpse: Continuous, Real-Time Object Recognition on Mobile Devices
Chen, T. Y.-H., Ravindranath, L., Deng, S., Bahl, P., & Balakrishnan, H. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809711 · Cited by 358 · cited 2×
“[119], [120], [121], [122], [123] Computation Offloading Based Edge Inference Systems Server-based edge inference: Partial data transmission for communication-efficient inference [124], [125], [126], [127] Raw data encoding for communication-efficient inference [128], [129] Cooperative downlink transmission for communica…”
BackgroundMoDNN: Local distributed mobile computing system for Deep Neural Network
Mao, J., Chen, X., Nixon, K. W., Krieger, C., & Chen, Y. (2017) · Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017 · doi:10.23919/date.2017.7927211 · Cited by 331
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed LearningPreprint
Chen, T., Giannakis, G. B., Sun, T., & Yin, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09965 · Cited by 329
Bayesian Compression for Deep Learning
Louizos, C., Ullrich, K., & Welling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08665 · Cited by 312
An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections
Cheng, Y., Yu, F. X., Feris, R. S., Kumar, S., Choudhary, A., & Chang, S.-F. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.327 · Cited by 303
Parallel stochastic gradient algorithms for large-scale matrix completion
Recht, B., & Ré, C. (2013) · Mathematical Programming Computation · doi:10.1007/s12532-013-0053-8 · Cited by 300
Device Placement Optimization with Reinforcement Learning
Mirhoseini, A., Pham, H., Le, Q. V., Steiner, B., Larsen, R., Zhou, Y., Kumar, N., Norouzi, M., Bengio, S., & Dean, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04972 · Cited by 299
SystemML: Declarative machine learning on MapReduce
Ghoting, A., Krishnamurthy, R., Pednault, E., Reinwald, B., Sindhwani, V., Tatikonda, S., Tian, Y., & Vaithyanathan, S. (2011) · 2011 IEEE 27th International Conference on Data Engineering · doi:10.1109/icde.2011.5767930 · Cited by 295
Communication-efficient algorithms for statistical optimization
Zhang, Y., Duchi, J. C., & Wainwright, M. J. (2012) · 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2012.6426691 · Cited by 283
A Linearly-Convergent Stochastic L-BFGS AlgorithmPreprint
Moritz, P., Nishihara, R., & Jordan, M. I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.02087 · Cited by 279
MIMO Over-the-Air Computation for High-Mobility Multimodal Sensing
Zhu, G., & Huang, K. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2871070 · Cited by 273
Privacy-preserving distributed mining of association rules on horizontally partitioned data
Kantarcıoğlu, M., & Clifton, C. (2004) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2004.45 · Cited by 262
Zhou, F., & Cong, G. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/447 · Cited by 255
DiSCO: Distributed Optimization for Self-Concordant Empirical Loss
Zhang, Y., & Xiao, L. (2015) · International Conference on Machine Learning · Cited by 243
Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
Chen, Y., Su, L., & Xu, J. (2018) · Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems · doi:10.1145/3219617.3219655 · Cited by 228
Toward a theory of in-network computation in wireless sensor networks
Giridhar, A., & Kumar, P. R. (2006) · IEEE Communications Magazine · doi:10.1109/mcom.2006.1632656 · Cited by 220
Yang, Z., Moczulski, M., Denil, M., De Freitas, N., Song, L., & Wang, Z. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.173 · Cited by 215 · cited 2×
“In [83], the authors presented an adaptive Fastfood transform to reparameterize the matrix-vector multiplication of fully-connected layers, thereby reducing the storage and computational costs.”
MethodCommunication Compression for Decentralized Training
Tang, H., Gan, S., Zhang, C., Zhang, T., & Liu, J. (2018) · arXiv · doi:10.48550/arxiv.1803.06443 · Cited by 206
Rigamonti, R., Sironi, A., Lepetit, V., & Fua, P. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.355 · Cited by 205
Communication-Computation Efficient Gradient CodingPreprint
Ye, M., & Abbe, E. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03475 · Cited by 203
BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems
Shao, J., & Zhang, J. (2020) · 2020 IEEE International Conference on Communications Workshops (ICC Workshops) · doi:10.1109/iccworkshops49005.2020.9145068 · Cited by 200
Self-citationGradient Sparsification for Communication-Efficient Distributed Optimization
Wangni, J., Wang, J., Ji, L. H., & Zhang, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.09854 · Cited by 198
Communication-efficient algorithms for statistical optimization
(2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2567769 · Cited by 195
Privacy-Preserving SVM Classification on Vertically Partitioned Data
Yu, H., Vaidya, J., & Jiang, X. (2006) · Lecture Notes in Computer Science · doi:10.1007/11731139_74 · Cited by 192
Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools
Mayer, R., & Jacobsen, H.-A. (2021) · ACM Computing Surveys · doi:10.1145/3363554 · Cited by 192
Not Just Privacy: Improving Performance of Private Deep Learning in Mobile Cloud
Wang, J., Zhang, J., Bao, W., Zhu, X., Cao, B., & Yu, P. S. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220106 · Cited by 188
Self-citationHarnessing Interference for Analog Function Computation in Wireless Sensor Networks
Goldenbaum, M., Boche, H., & Stanczak, S. (2013) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2013.2272921 · Cited by 183
Gradient Coding From Cyclic MDS Codes and Expander Graphs
Raviv, N., Tamo, I., Tandon, R., & Dimakis, A. G. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.3029396 · Cited by 179
Communication-Efficient Distributed Dual Coordinate Ascent
Jaggi, M., Smith, V., Takáč, M., Terhorst, J., Krishnan, S., Hofmann, T., & Jordan, M. I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.1458 · Cited by 177
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDPreprint
Wang, J., & Joshi, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08313 · Cited by 175
Net-Trim: Convex Pruning of Deep Neural Networks with Performance GuaranteePreprint
Aghasi, A., Abdi, A., Nguyen, N. H., & Romberg, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05162 · Cited by 173
Gradient Sparsification for Communication-Efficient Distributed Optimization
Wangni, J., Wang, J., Liu, J., & Zhang, T. (2018) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 166
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
Chen, C.-Y., Choi, J., Brand, D., Agrawal, A., Zhang, W., & Gopalakrishnan, K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11728 · Cited by 166
Improving Distributed Gradient Descent Using Reed-Solomon Codes
Halbawi, W., Azizan, N., Salehi, F., & Hassibi, B. (2018) · 2018 IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2018.8437467 · Cited by 157
Cloud-Based Collaborative 3D Mapping in Real-Time With Low-Cost Robots
Mohanarajah, G., Usenko, V., Singh, M., D’Andrea, R., & Waibel, M. (2015) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2015.2408456 · Cited by 155
Ko, J. H., Na, T., Amir, M. F., & Mukhopadhyay, S. (2018) · 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2018.8639121 · Cited by 153
Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squaresPreprint
Pilancı, M., & Wainwright, M. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.0347 · Cited by 147
Wang, J., & Joshi, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07576 · Cited by 146
Privacy-Preserving Decision Trees over Vertically Partitioned Data
Vaidya, J., & Clifton, C. (2005) · Lecture Notes in Computer Science · doi:10.1007/11535706_11 · Cited by 144
Coding for Distributed Fog Computing
Li, S., Maddah-Ali, M. A., & Avestimehr, A. S. (2017) · IEEE Communications Magazine · doi:10.1109/mcom.2017.1600894 · Cited by 138
ZONE: Zeroth-Order Nonconvex Multiagent Optimization Over Networks
Hajinezhad, D., Hong, M., & Garcia, A. (2019) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2019.2896025 · Cited by 137
Distributed Mean Estimation with Limited Communication
Suresh, A. T., Yu, F. X., Kumar, S., & McMahan, H. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00429 · Cited by 135
Communication Complexity of Distributed Convex Learning and Optimization
Arjevani, Y., & Shamir, O. (2015) · arXiv · doi:10.48550/arxiv.1506.01900 · Cited by 133 · cited 1×
“Arjevani and Shamir [164] studied the scenarios where each device has access to a different subset of data to minimize the averaged loss function over all devices.”
BackgroundA Scalable Framework for Wireless Distributed Computing
Li, S., Yu, Q., Maddah-Ali, M. A., & Avestimehr, A. S. (2017) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2017.2702605 · Cited by 130
Compressing Convolutional Neural Networks in the Frequency Domain
Chen, W., Wilson, J., Tyree, S., Weinberger, K. Q., & Chen, Y. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939839 · Cited by 127
DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework
Liu, Z., Liu, T., Wen, W., Jiang, L., Xu, J., Wang, Y., & Quan, G. (2018) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3195970.3196022 · Cited by 124
Randomized Gradient-Free Method for Multiagent Optimization Over Time-Varying Networks
Yuan, D., & Ho, D. W. C. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2336806 · Cited by 121
GIANT: Globally Improved Approximate Newton Method for Distributed OptimizationPreprint
Wang, S., Roosta, F., Xu, P., & Mahoney, M. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.03528 · Cited by 121
Distributed stochastic optimization and learning
Shamir, O., & Srebro, N. (2014) · 2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton) · doi:10.1109/allerton.2014.7028543 · Cited by 120 · cited 1×
“Shamir and Srebro [159] proposed to accelerate mini-batch SGD by using the largest possible mini-batch size that does not hurt the sample complexity, and it shows that the communication cost decreases linearly with the size of the mini-batch.”
BackgroundGradient Diversity: a Key Ingredient for Scalable Distributed LearningPreprint
Yin, D., Pananjady, A., Lam, M. W. Y., Papailiopoulos, D., Ramchandran, K., & Bartlett, P. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05699 · Cited by 119
Network offloading policies for cloud robotics: a learning-based approach
Chinchali, S., Sharma, A., Harrison, J., Elhafsi, A., Kang, D., Pergament, E., Cidon, E., Katti, S., & Pavone, M. (2021) · Autonomous Robots · doi:10.1007/s10514-021-09987-4 · Cited by 117
Communication-constrained multi-AUV cooperative SLAM
Paull, L., Huang, G., Seto, M., & Leonard, J. J. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7139227 · Cited by 116
High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning
Du, Y., Yang, S., & Huang, K. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.2983166 · Cited by 116
Sironi, A., Tekin, B., Rigamonti, R., Lepetit, V., & Fua, P. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2343229 · Cited by 110
GradientFlow: Optimizing Network Performance for Large-Scale Distributed DNN Training
Sun, P., Wen, Y., Han, R., Feng, W., & Yan, S. (2019) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2957478 · Cited by 109
SketchML: Accelerating Distributed Machine Learning with Data Sketches
Jiang, J., Fu, F., Yang, T., & Cui, B. (2018) · Proceedings of the 2018 International Conference on Management of Data · doi:10.1145/3183713.3196894 · Cited by 102
Why random reshuffling beats stochastic gradient descent
Gürbüzbalaban, M., Ozdaglar, A., & Parrilo, P. A. (2021) · Mathematical Programming · doi:10.1007/s10107-019-01440-w · Cited by 101
Smith, V., Chiang, C.-K., Sanjabi, M., & Talwalkar, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10467 · Cited by 101
Near-Optimal Straggler Mitigation for Distributed Gradient Methods
Li, S., Mousavi Kalan, S. M., Avestimehr, A. S., & Soltanolkotabi, M. (2018) · 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) · doi:10.1109/ipdpsw.2018.00137 · Cited by 98
Tactile Robots as a Central Embodiment of the Tactile Internet
Haddadin, S., Johannsmeier, L., & Ledezma, F. D. (2018) · Proceedings of the IEEE · doi:10.1109/jproc.2018.2879870 · Cited by 95
Structured Transforms for Small-Footprint Deep Learning
Sindhwani, V., Sainath, T. N., & Kumar, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.01722 · Cited by 93
Secure Linear Regression on Vertically Partitioned Datasets.Preprint
Gascón, A., Schoppmann, P., Balle, B., Raykova, M., Doerner, J., Zahur, S., & Evans, D. (2016) · IACR Cryptology ePrint Archive · Cited by 92
Wireless Network Intelligence at the Edge
Park, J., Samarakoon, S., Bennis, M., & Debbah, M. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2941458 · Cited by 92
Yu, M., Lin, Z., Narra, K. G., Li, S., Li, Y., Kim, N. S., Schwing, A. G., Annavaram, M., & Avestimehr, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03617 · Cited by 89
Stochastic Gradient Coding for Straggler Mitigation in Distributed Learning
Bitar, R., Wootters, M. K., & Rouayheb, S. E. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2991361 · Cited by 86
How to scale distributed deep learning?
Jin, P. H., Yuan, Q., Iandola, F., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1611.04581 · Cited by 85 · cited 2×
“This approach has been adopted in distributed deep learning for model updating [110], [111].”
BackgroundMethodImproving Device-Edge Cooperative Inference of Deep Learning via 2-Step Pruning
Shi, W., Hou, Y., Zhou, S., Niu, Z., Zhang, Y., & Geng, L. (2019) · IEEE INFOCOM 2019 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infocomwkshps47286.2019.9093772 · Cited by 85
Speeding Up Distributed Gradient Descent by Utilizing Non-persistent Stragglers
Özfatura, E., Gündüz, D., & Ulukuş, Ş. (2019) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849684 · Cited by 82
GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
Daily, J., Vishnu, A., Siegel, C., Warfel, T., & Amatya, V. (2018) · arXiv · doi:10.48550/arxiv.1803.05880 · Cited by 80 · cited 14×
“In edge AI systems with inherently isolated data, distributed realizations of SGD will play a key role and should be carefully investigated.”
BackgroundMethodInfluentialDistributed stochastic variance reduced gradient methods by sampling extra data with replacement
Lee, J. D., Lin, Q., Ma, T., & Yang, T. (2017) · Journal of Machine Learning Research · Cited by 78
Gossip training for deep learning
Blot, M., Picard, D., Cord, M., & Thome, N. (2016) · arXiv · doi:10.48550/arxiv.1611.09726 · Cited by 77 · cited 2×
“…mitigation via coded computing [93], [94], [95], [96], [97], [98], [99], [100], [101] • Training in decentralized system mode [102], [103], [104], [105], [106], [107], [108], [109], [110], [111], [112] Model Partition Based Edge Training Systems • Model partition across a large number of nodes to…”
MethodCommunication-Efficient Distributed Optimization of Self-concordant Empirical Loss
Zhang, Y., & Xiao, L. (2018) · Lecture notes in mathematics · doi:10.1007/978-3-319-97478-1_11 · Cited by 76
The Role of Network Topology for Distributed Machine Learning
Neglia, G., Calbi, G., Towsley, D., & Vardoyan, G. (2019) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2019.8737602 · Cited by 76
Reconfigurable Intelligent Surface for Green Edge Inference
Hua, S., Zhou, Y., Yang, K., Shi, Y., & Wang, K. (2021) · IEEE Transactions on Green Communications and Networking · doi:10.1109/tgcn.2021.3058657 · Cited by 74 · cited 3×
“Fortunately, offloading the task from edge devices to powerful edge servers emerges as an antidote [13], [130], [131], [133].”
BackgroundMethodSelf-citationSoft Weight-Sharing for Neural Network Compression
Ullrich, K., Meeds, E., & Welling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04008 · Cited by 72
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed TrainingPreprint
Chen, C. Y., Choi, J., Brand, D., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.02679 · Cited by 71
On the classification of fog computing applications: A machine learning perspective
Guevara, J. C., Torres, R. D. S., & da Fonseca, N. L. S. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2020.102596 · Cited by 69
Variance-Reduced Stochastic Learning by Networked Agents Under Random Reshuffling
Yuan, K., Ying, B., Liu, J., & Sayed, A. H. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2018.2872003 · Cited by 67
Blind Over-the-Air Computation and Data Fusion via Provable Wirtinger Flow
Dong, J., Shi, Y., & Ding, Z. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.2970338 · Cited by 65
Self-citationEnergy-Efficient Processing and Robust Wireless Cooperative Transmission for Edge Inference
Yang, K., Shi, Y., Yu, W., & Ding, Z. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2979523 · Cited by 65
Self-citationRobust Gradient Descent via Moment Encoding and LDPC Codes
Maity, R. K., Rawat, A. S., & Mazumdar, A. (2019) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849514 · Cited by 65
Zeroth-Order Method for Distributed Optimization With Approximate Projections
Yuan, D., Ho, D. W. C., & Xu, S. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2480419 · Cited by 56
Distributed optimization for deep learning with gossip exchange
Blot, M., Picard, D., Thome, N., & Cord, M. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.11.002 · Cited by 55
Communication-Efficient Distributed Optimization of Self-Concordant Empirical Loss
Zhang, Y., & Xiao, L. (2015) · arXiv · doi:10.48550/arxiv.1501.00263 · Cited by 55
A hybrid approach to offloading mobile image classification
Hauswald, J., Manville, T., Zheng, Q., Dreslinski, R., Chakrabarti, C., & Mudge, T. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6855235 · Cited by 54
Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT
Bhardwaj, K., Lin, C.-Y., Sartor, A., & Marculescu, R. (2019) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3358205 · Cited by 53
A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression
Yang, K., Fan, T., Chen, T., Shi, Y., & Yang, Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.00513 · Cited by 52
Self-citationSahu, A. K., Jakovetić, D., Bajović, D., & Kar, S. (2018) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2018.8619044 · Cited by 51
Learning a Code: Machine Learning for Approximate Non-Linear Coded ComputationPreprint
Kosaian, J., Rashmi, K. V., & Venkataraman, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01259 · Cited by 48
Neural Networks Meet Physical Networks: Distributed Inference Between Edge Devices and the Cloud
Chinchali, S. P., Cidon, E., Pergament, E., Chu, T., & Katti, S. (2018) · Proceedings of the 17th ACM Workshop on Hot Topics in Networks · doi:10.1145/3286062.3286070 · Cited by 46
Sun, P., Feng, W., Han, R., Yan, S., & Wen, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06855 · Cited by 46
Straggler-Aware Distributed Learning: Communication–Computation Latency Trade-Off
Özfatura, E., Ulukuş, Ş., & Gündüz, D. (2020) · Entropy · doi:10.3390/e22050544 · Cited by 46
Data Shuffling in Wireless Distributed Computing via Low-Rank Optimization
Yang, K., Shi, Y., & Ding, Z. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2019.2912139 · Cited by 43 · cited 6×
“Interestingly, coding technique plays a critical role in scalable data shuffling [31], [141] as well as straggler mitigation [145].”
BackgroundMethodInfluentialSelf-citationCommunication-efficient Algorithms for Distributed Stochastic Principal Component AnalysisPreprint
Garber, D., Shamir, O., & Srebro, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08169 · Cited by 43
Latency analysis of coded computation schemes over wireless networks
Reisizadeh, A., & Pedarsani, R. (2017) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2017.8262881 · Cited by 43
Research works printed but not linked: 10
- 2016, Regulation (EU) 2016/679 of the European parliament and of the council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (general data protection regulation), Official J Eur Union, 119, 1
- bughin, 2018, Assessing the Economic Impact of Artificial Intelligence
- chen, 2019, signSGD via zeroth-order oracle, Proc Int Conf Learn Represent (ICLR), 6
- 2018, Cisco Global Cloud Index Forecast and Methodology 2016–2021 White Paper
- 10.1109/twc.2019.2963654 · doi:10.1109/twc.2019.2963654
- 10.15607/rss.2019.xv.063 · doi:10.15607/rss.2019.xv.063
- 2019, NVIDIA Clara An Application Framework Optimized for Healthcare and Life Sciences Developers
- 2018, Fate An Industrial Grade Federated Learning Framework
- hu, 2015, Mobile edge computing—A key technology towards 5G
- mao, 2018, A privacy-preserving deep learning approach for face recognition with edge computing, Proc USENIX Workshop Hot Topics Edge Comput (HotEdge), 308
Cited by
Federated learning on non-IID data: A survey
Zhu, H., Xu, J., Liu, S., & Jin, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.098 · Cited by 1,062
Edge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“edge devices are essential to reduce the uplink communication overheads, for which various data dimensionality reduction approaches have been proposed by exploiting the specific computation tasks and communication environments [29].”
BackgroundMethodInfluentialSelf-citationAgentic AI: Autonomous intelligence for complex goals–a comprehensive survey
Acharya, D. B., Kuppan, K., & Divya, B. (2025) · IEEE Access · doi:10.1109/access.2025.3532853 · Cited by 717
Siriwardhana, Y., Porambage, P., Liyanage, M., & Ylianttila, M. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3061981 · Cited by 674
Singh, R., & Gill, S. S. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.02.004 · Cited by 572
Metadata
- Authors
- Shi, Y., Yang, K., Jiang, T., Zhang, J., & Letaief, K. B.
- Affiliations
- ShanghaiTech University (CN)
- Chinese Academy of Sciences (CN)
- Hong Kong Polytechnic University (HK)
- Venue
- IEEE Communications Surveys & Tutorials
- Published
- 2020
- Volume
- 22
- Issue
- 4
- Pages
- 2167-2191
- ISSN
- 1553-877X (ISSN-L)
- 2373-745X
- OpenAlex
- W3041971333
- Cited by
- 488
- References
- 237
- Citations by year
- 2026: 58 · 2025: 97 · 2024: 93 · 2023: 100 · 2022: 73 · 2021: 53 · 2020: 12
- FWCI
- 27.96
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Communications Surveys & Tutorials (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Nature Science Foundation of China (61601290)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- edge AI, communication efficiency, edge computing, communication overhead, distributed learning, edge inference, IoT devices, privacy, energy consumption, wireless networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.