Dynamic Adaptive DNN Surgery for Inference Acceleration on the Edge
The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Abstract
Recent advances in deep neural networks (DNNs) have substantially improved the accuracy and speed of a variety of intelligent applications. Nevertheless, one obstacle is that DNN inference imposes heavy computation burden to end devices, but offloading inference tasks to the cloud causes transmission of a large volume of data. Motivated by the fact that the data size of some intermediate DNN layers is significantly smaller than that of raw input data, we design the DNN surgery, which allows partitioned DNN processed at both the edge and cloud while limiting the data transmission. The challenge is twofold: (1) Network dynamics substantially influence the performance of DNN partition, and (2) State-of-the-art DNNs are characterized by a directed acyclic graph (DAG) rather than a chain so that partition is greatly complicated. In order to solve the issues, we design a Dynamic Adaptive DNN Surgery (DADS) scheme, which optimally partitions the DNN under different network condition. Under the lightly loaded condition, DNN Surgery Light (DSL) is developed, which minimizes the overall delay to process one frame. The minimization problem is equivalent to a min-cut problem so that a globally optimal solution is derived. In the heavily loaded condition, DNN Surgery Heavy (DSH) is developed, with the objective to maximize throughput. However, the problem is NP-hard so that DSH resorts an approximation method to achieve an approximation ratio of 3. Real-world prototype based on self-driving car video dataset is implemented, showing that compared with executing entire the DNN on the edge and cloud, DADS can improve latency up to 6.45 and 8.08 times respectively, and improve throughput up to 8.31 and 14.01 times respectively.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892 · cited 4×
“For example, the input data size of tiny YOLOv2 [4] is 0 .”
BackgroundInfluentialInception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · cited 1×
“6 shows the DAG of the pure inception v4 network [10] in Fig.”
BackgroundAn experimental comparison of min-cut/max- flow algorithms for energy minimization in vision
Boykov, Y., & Kolmogorov, V. (2004) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2004.60 · Cited by 4,577
MAUI: making smartphones last longer with code offload
Cuervo, E., Balasubramanian, A., Cho, D.-K., Wolman, A., Saroiu, S., Chandra, R., & Bahl, P. (2010) · Proceedings of the 8th international conference on Mobile systems, applications, and services · doi:10.1145/1814433.1814441 · Cited by 2,281
Dynamic adaptive streaming over HTTP -: standards and design principles
Stockhammer, T. (2011) · Proceedings of the second annual ACM conference on Multimedia systems · doi:10.1145/1943552.1943572 · Cited by 1,221 · cited 1×
“This can be realized by a method similar to HTTP DASH [11].”
MethodDeep neural networks for small footprint text-dependent speaker verification
Variani, E., Lei, X., McDermott, E., Moreno, I. L., & Gonzalez-Dominguez, J. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854363 · Cited by 1,035
Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices
Teerapittayanon, S., McDanel, B., & Kung, H. T. (2017) · 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2017.226 · Cited by 843
(2011) · Cited by 713
Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge
Kang, Y., Hauswald, J., Gao, C., Rovinski, A., Mudge, T., Mars, J., & Tang, L. (2017) · Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3037697.3037698 · Cited by 673
DeepX: A Software Accelerator for Low-Power Deep Learning Inference on Mobile Devices
Lane, N. D., Bhattacharya, S., Georgiev, P., Forlivesi, C., Jiao, L., Qendro, L., & Kawsar, F. (2016) · 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn.2016.7460664 · Cited by 450 · cited 1×
“DeepX [23] explored the opportunities to use mobile GPUs to enable real-time deep learning inferences.”
BackgroundHan, S., Shen, H., Philipose, M., Agarwal, S., Wolman, A., & Krishnamurthy, A. (2016) · Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2906388.2906396 · Cited by 367
(2017) · Networked Systems Design and Implementation · Cited by 329
Beyond throughput: a 4G LTE dataset with channel and context metrics
Raca, D., Quinlan, J. J., Zahran, A. H., & Sreenan, C. J. (2018) · Proceedings of the 9th ACM Multimedia Systems Conference · doi:10.1145/3204949.3208123 · Cited by 216
DeepBurning: automatic generation of FPGA-based learning accelerators for the neural network family
Wang, Y., Xu, J., Han, Y., Li, H., & Li, X. (2016) · Proceedings of the 53rd Annual Design Automation Conference · doi:10.1145/2897937.2898003 · Cited by 205
Millimeter Wave Vehicular Communications: A Survey
Va, V., Shimizu, T., Bansal, G., & Heath, R. W. (2016) · Foundations and Trends in Networking · doi:10.1561/1300000054 · Cited by 205
Adaptive deep learning model selection on embedded systems
Taylor, B., Marco, V. S., Wolff, W., Elkhatib, Y., & Wang, Z. (2018) · Proceedings of the 19th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems · doi:10.1145/3211332.3211336 · Cited by 90
Bringing contextual information to google speech recognition
Aleksic, P. S., Ghodsi, M., Michaely, A. H., Allauzen, C., Hall, K., Brian Roark, A., Rybach, D., & Moreno, P. J. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-177 · Cited by 90
Measuring TCP over WiFi: A Real Case
Franceschinis, M., Mellia, M., Meo, M., & Munafò, M. M. (2005) · PORTO Publications Open Repository TOrino (Politecnico di Torino) · Cited by 22
Bonsma, P. (2009) · Discrete Applied Mathematics · doi:10.1016/j.dam.2009.09.010 · Cited by 22
Adaptive Selection of Deep Learning Models on Embedded SystemsPreprint
Taylor, B., Marco, V. S., Wolff, W., Elkhatib, Y., & Wang, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04252 · Cited by 6
Cited by
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455
“[80] propose an approximation algorithm that provides worst-case performance guarantee, based on the graph min-cut method.”
MethodModel Pruning Enables Efficient Federated Learning on Edge Devices
Jiang, Y., Wang, S., Valls, V., Ko, B. J., Lee, W.-H., Leung, K. K., & Tassiulas, L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3166101 · Cited by 589
“, knowledge distillation [36], runtime neural pruning [37], and DNN partitioning and offloading [38], do not serve our purpose.”
BackgroundUnleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services
Xu, M., Du, H., Niyato, D., Kang, J., Xiong, Z., Mao, S., Han, Z., Jamalipour, A., Kim, D. I., Shen, X., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3353265 · Cited by 449
“AIGC service provisioning based on edge intelligence is computationally and communication-intensive for resourceconstrained edge servers and mobile devices [132].”
Holistic Network Virtualization and Pervasive Network Intelligence for 6G
Shen, X., Gao, J., Wu, W., Li, M., Zhou, C., & Zhuang, W. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3135829 · Cited by 392
SPINN: synergistic progressive inference of neural networks over device and cloud
Laskaridis, S., Venieris, S. I., Almeida, M., Leontiadis, I., & Lane, N. D. (2020) · Proceedings of the 26th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3372224.3419194 · Cited by 310
“on [34], partitions the CNN between a device-mapped head and a cloud-mapped tail and selects a single split point based on the device and cloud load as well as the network conditions. Similarly, DADS [22] tackles CNN offloading, but from a scheduler-centric standpoint, with the aim to yield the optimal partitioning sc…”
Background
Metadata
- Authors
- Hu, C., Bao, W., Wang, D., & Liu, F.
- Affiliations
- Hong Kong Polytechnic University (HK)
- The University of Sydney (AU)
- Venue
- IEEE INFOCOM 2019 - IEEE Conference on Computer Communications
- Publisher
- IEEE
- Published
- 2019-04
- Pages
- 1423-1431
- OpenAlex
- W2920031528
- Cited by
- 416
- References
- 23
- Citations by year
- 2026: 29 · 2025: 61 · 2024: 65 · 2023: 92 · 2022: 62 · 2021: 72 · 2020: 26 · 2019: 9
- FWCI
- 36.52
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Conference on Computer Communications (INFOCOM) (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Cloud Computing and Remote Desktop Technologies
- Keywords
- DNN partitioning, cloud-edge collaboration, inference acceleration, dynamic network conditions, directed acyclic graphs, minimum cut, approximation algorithms, throughput maximization, latency reduction, data transmission reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.