JALAD: Joint Accuracy-And Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution
Open accessThe DOI matches its Crossref record; 44 of the 45 references it cites were found in a registry or an index.
Abstract
Recent years have witnessed a rapid growth of deep-network based services and applications. A practical and critical problem thus has emerged: how to effectively deploy the deep neural network models such that they can be executed efficiently. Conventional cloud-based approaches usually run the deep models in data center servers, causing large latency because a significant amount of data has to be transferred from the edge of network to the data center. In this paper, we propose JALAD, a joint accuracy- and latency-aware execution framework, which decouples a deep neural network so that a part of it will run at edge devices and the other part inside the conventional cloud, while only a minimum amount of data has to be transferred between them. Though the idea seems straightforward, we are facing challenges including i)how to find the best partition of a deep structure; ii)how to deploy the component at an edge device that only has limited computation power; and iii)how to minimize the overall execution latency. Our answers to these questions are a set of strategies in JALAD, including 1)A normalization based in-layer data compression strategy by jointly considering compression rate and model accuracy; 2)A latency-aware deep decoupling strategy to minimize the overall execution latency; and 3)An edge-cloud structure adaptation strategy that dynamically changes the decoupling for different network conditions. Experiments demonstrate that our solution can significantly reduce the execution latency: it speeds up the overall inference execution with a guaranteed model accuracy loss.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 7×
“We decoupled VGGNet (16 layers and 19 layers version), and ResNet (50 layers and 101 layers version) to evaluate JALAD’s execution speedup and accuracy performance.”
MethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 6×
“We decoupled VGGNet (16 layers and 19 layers version), and ResNet (50 layers and 101 layers version) to evaluate JALAD’s execution speedup and accuracy performance.”
MethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“In traditional sequential DNNs like AlexNet [15], VGGNet, input signals flow straightforward layer by layer.”
MethodGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 6×
“To verify the compression performance, we compress the in-layer feature maps of 5000 samples from ILSVRC2012 validation set.”
MethodInfluentialRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Edge computing: Vision and challenges
Shi, W., Cao, J., Zhang, Q., Li, Y., & Xu, L. (2016) · IEEE Internet of Things Journal · doi:10.1109/jiot.2016.2579198 · Cited by 8,733
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · cited 1×
“Deep neural network has become the de facto structure for today’s practical machine learning model selection [22], thanks to its simplicity and effectiveness.”
BackgroundFog computing and its role in the internet of things
Bonomi, F., Milito, R., Zhu, J., & Addepalli, S. (2012) · Proceedings of the first edition of the MCC workshop on Mobile cloud computing · doi:10.1145/2342509.2342513 · Cited by 6,244
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 1×
“quantization method has been proved feasible for compressing the parameters [18], our experiments show that it also applies to compress feature values.”
MethodThe case for vm-based cloudlets in mobile computing
Satyanarayanan, M., Bahl, P., Caceres, R., & Davies, N. (2009) · IEEE Pervasive Computing · doi:10.1109/mprv.2009.82 · Cited by 3,660
MAUI: making smartphones last longer with code offload
Cuervo, E., Balasubramanian, A., Cho, D.-K., Wolman, A., Saroiu, S., Chandra, R., & Bahl, P. (2010) · Proceedings of the 8th international conference on Mobile systems, applications, and services · doi:10.1145/1814433.1814441 · Cited by 2,281 · cited 1×
“Traditional solution to deal with the trade-off is offloading computation from client [37]–[39], but these approaches are designed for generalized application, did not focus on DNN’s special structure.”
BackgroundInteger Programming with a Fixed Number of Variables
Lenstra, H. W. (1983) · Mathematics of Operations Research · doi:10.1287/moor.8.4.538 · Cited by 1,440 · cited 1×
“In the above ILP problem, the number of variable xic is equal to N ·C and therefore fixed and not too large, this fixed variable-amount ILP problem can be solved in polynomial time [21].”
BackgroundA Survey of Fog Computing: Concepts, Applications and Issues
Yi, S., Li, C., & Li, Q. (2015) · Proceedings of the 2015 Workshop on Mobile Big Data · doi:10.1145/2757384.2757397 · Cited by 1,339 · cited 1×
“, cloud computing, fog computing, edge computing) naturally becomes the choice for supporting deep-structurebased services and applications [23]–[27].”
BackgroundMobile edge computing, Fog et al.: A survey and analysis of security threats and challenges
Roman, R., Lopez, J., & Mambo, M. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2016.11.009 · Cited by 1,338
Compressing Deep Convolutional Networks using Vector Quantization
Gong, Y., Liu, L., Yang, M., & Bourdev, L. D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6115 · Cited by 925 · cited 1×
“Another solution is to run a mini-version of the original deep network models on edge devices, usually a compressed version of the original deep network model exported by pruning [5]– [7] or quantization [8]–[10].”
BackgroundDistributed Deep Neural Networks Over the Cloud, the Edge and End Devices
Teerapittayanon, S., McDanel, B., & Kung, H. T. (2017) · 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2017.226 · Cited by 843
DeepTrust^RT: Confidential Deep Neural Inference Meets Real-Time!
Babar, M. F., & Hasan, M. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.ecrts.2024.13 · Cited by 708
Structured Pruning of Deep Convolutional Neural Networks
Anwar, S., Hwang, K., & Sung, W. (2017) · ACM Journal on Emerging Technologies in Computing Systems · doi:10.1145/3005348 · Cited by 698 · cited 1×
“Another solution is to run a mini-version of the original deep network models on edge devices, usually a compressed version of the original deep network model exported by pruning [5]– [7] or quantization [8]–[10].”
BackgroundNeurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge
Kang, Y., Hauswald, J., Gao, C., Rovinski, A., Mudge, T., Mars, J., & Tang, L. (2017) · Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3037697.3037698 · Cited by 673
Towards wearable cognitive assistance
Ha, K., Chen, Z., Hu, W., Richter, W., Pillai, P., & Satyanarayanan, M. (2014) · Proceedings of the 12th annual international conference on Mobile systems, applications, and services · doi:10.1145/2594368.2594383 · Cited by 476 · cited 2×
“Edge-based deployment: An alternative is to deploy deep neural networks at the edge of network [30], [31].”
BackgroundDeepDecision: A Mobile Deep Learning Framework for Edge Video Analytics
Ran, X., Chen, H., Zhu, X., Liu, Z., & Chen, J. (2018) · IEEE INFOCOM 2018 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2018.8485905 · Cited by 450 · cited 1×
“Another work [44] proposed to adaptively upload compressed data or deploy compressed model locally to jointly execute, but they didn’t look into and modify the deep model itself.”
BackgroundDeepX: A Software Accelerator for Low-Power Deep Learning Inference on Mobile Devices
Lane, N. D., Bhattacharya, S., Georgiev, P., Forlivesi, C., Jiao, L., Qendro, L., & Kawsar, F. (2016) · 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn.2016.7460664 · Cited by 450 · cited 1×
“In [32], [33], Lane et al. first explored the feasibility of running deep neural networks on mobile devices for audio-sensing applications.”
BackgroundHan, S., Shen, H., Philipose, M., Agarwal, S., Wolman, A., & Krishnamurthy, A. (2016) · Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2906388.2906396 · Cited by 367 · cited 1×
“[41] implemented a streaming analysis system that balances the classification accuracy and resource demand by scheduling model variants between resource-constrained devices and cloud.”
MethodGlimpse: Continuous, Real-Time Object Recognition on Mobile Devices
Chen, T. Y.-H., Ravindranath, L., Deng, S., Bahl, P., & Balakrishnan, H. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809711 · Cited by 358 · cited 1×
“To reduce the latency, [28] proposed a bandwidth efficiency object tracking system by dropping video frames from the raw video to the cloud; Jain et al.”
BackgroundDeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications
Huynh, L. N., Lee, Y., & Balan, R. K. (2017) · Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3081333.3081360 · Cited by 329 · cited 1×
“[35] optimized the execution of deep learning models on mobile GPUs.”
BackgroundDeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning
Lane, N. D., Georgiev, P., & Qendro, L. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2804262 · Cited by 310 · cited 1×
“In [32], [33], Lane et al. first explored the feasibility of running deep neural networks on mobile devices for audio-sensing applications.”
BackgroundCollaborative Task Execution in Mobile Cloud Computing Under a Stochastic Wireless Channel
Zhang, W., Wen, Y., & Wu, D. O. (2015) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2014.2331051 · Cited by 247 · cited 1×
“Traditional solution to deal with the trade-off is offloading computation from client [37]–[39], but these approaches are designed for generalized application, did not focus on DNN’s special structure.”
BackgroundSelf-citationFixed-point feedforward deep neural network design using weights +1, 0, and −1
Hwang, K., & Sung, W. (2014) · 2014 IEEE Workshop on Signal Processing Systems (SiPS) · doi:10.1109/sips.2014.6986082 · Cited by 244
Fixed point optimization of deep convolutional neural networks for object recognition
Anwar, S., Hwang, K., & Sung, W. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178146 · Cited by 239 · cited 1×
“Another solution is to run a mini-version of the original deep network models on edge devices, usually a compressed version of the original deep network model exported by pruning [5]– [7] or quantization [8]–[10].”
BackgroundThe Case for VM-based Cloudlets in Mobile Computing
Satyanarayanan, M., Bahl, V., Caceres, R., & Davies, N. (2011) · IEEE Pervasive Computing · doi:10.1109/mprv.2009.64 · Cited by 239
Fog Computing Based Face Identification and Resolution Scheme in Internet of Things
Hu, P., Ning, H., Qiu, T., Zhang, Y., & Luo, X. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2016.2607178 · Cited by 209 · cited 2×
“[34] proposed face identification deployment framework based on a fog computing infrustructure.”
BackgroundMethodNeurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge
Kang, Y., Hauswald, J., Gao, C., Rovinski, A., Mudge, T., Mars, J., & Tang, L. (2017) · ACM SIGPLAN Notices · doi:10.1145/3093336.3037698 · Cited by 185
A Berkeley View of Systems Challenges for AI
Stoica, I., Song, D., Popa, R. A., Patterson, D., Mahoney, M. W., Katz, R., Joseph, A. D., Jordan, M., Hellerstein, J. M., Gonzalez, J. E., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05855 · Cited by 171 · cited 1×
“, cloud computing, fog computing, edge computing) naturally becomes the choice for supporting deep-structurebased services and applications [23]–[27].”
BackgroundPaleo: A Performance Model for Deep Neural Networks
Qi, H., Sparks, E., & Talwalkar, A. (2017) · International Conference on Learning Representations · Cited by 154
Scalable Distributed Computing Hierarchy: Cloud, Fog and Dew Computing
Skala, K., Davidović, D., Afgan, E., Sović, I., & Šojat, Z. (2015) · Institutional Repository of the Ruđer Bošković Institute (Ruđer Bošković Institute) · doi:10.19210/1002.2.1.16 · Cited by 153 · cited 1×
“Cloud-based deployment: Conventionally, most of today’s deep neural network usually deployed on dedicated servers in the datacenter [26].”
BackgroundJust-in-time provisioning for cyber foraging
Ha, K., Pillai, P., Richter, W., Abe, Y., & Satyanarayanan, M. (2013) · Proceeding of the 11th annual international conference on Mobile systems, applications, and services · doi:10.1145/2462456.2464451 · Cited by 146 · cited 1×
“Edge-based deployment: An alternative is to deploy deep neural networks at the edge of network [30], [31].”
BackgroundOverLay: Practical Mobile Augmented Reality
Jain, P., Manweiler, J., & Roy Choudhury, R. (2015) · Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2742647.2742666 · Cited by 116 · cited 1×
“[29] proposed to make use of the blurred frames so as to reduce upload load.”
BackgroundEshratifar, A. E., & Pedram, M. (2018) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3194554.3194565 · Cited by 115 · cited 1×
“[43] presented a solution to decouple a deep structure into 3 segments without in-layer feature compression.”
Towards Wearable Cognitive Assistance
Ha, K., Chen, Z., Hu, W., Richter, W., Pillai, P. S., & Satyanarayanan, M. (2013) · Defense Technical Information Center (DTIC) · doi:10.21236/ada591470 · Cited by 94
Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices
Lu, Z., Rallapalli, S., Chan, K., & La Porta, T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123389 · Cited by 69 · cited 2×
“A parallel work [11] used a regression model to estimate the execution time of DNNs, [19] and [20] tried to give analytic estimation of DNNs’ execution time but failed to estimate fine-grained layer-level execution time.”
BackgroundMethodToward transcoding as a service: energy-efficient offloading policy for green mobile cloud
Zhang, W., Wen, Y., & Chen, H.-H. (2014) · IEEE Network · doi:10.1109/mnet.2014.6963807 · Cited by 57
Self-citationA hybrid approach to offloading mobile image classification
Hauswald, J., Manville, T., Zheng, Q., Dreslinski, R., Chakrabarti, C., & Mudge, T. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6855235 · Cited by 54 · cited 1×
“For current applications with deep neural network structure, some partition scheme [34], [40] tried to extract features locally to assist the classification DNN in the cloud.”
MethodA Deep Neural Network Compression Pipeline: Pruning, Quantization, Huffman EncodingPreprint
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv (Cornell University) · Cited by 37
Research works printed but not linked: 5
- roman, 2016, Mobile edge computing: a survey and analysis of security threats and challenges, arXiv preprint arXiv 1602 00484
- index, 0, Global mobile data traffic forecast update 2014-2019 white paper c11-520862
- han, 2015, Learning both weights and connections for efficient neural network, Advances in neural information processing systems, 1115
- 0, Apple moves to third-generation siri back-end built on open-source mesos platform
- 0, Google Supercharges Machine Learning Tasks with TPU Custom Chip
Cited by
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455
“[81] H. Li, C. Hu, J. Jiang, Z. Wang, Y. Wen, and W. Zhu, “JALAD: Joint accuracy-and latency-aware deep structure decoupling for edge-cloud execution,” 2018, arXiv:1812.10027.”
MethodInfluentialBeyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications
Gunduz, D., Qin, Z., Aguerri, I. E., Dhillon, H. S., Yang, Z., Yener, A., Wong, K. K., & Chae, C.-B. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3223408 · Cited by 776
“Quantization and/or compression of feature vectors is considered in [185], [186] and [187].”
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
Chen, M., Gunduz, D., Huang, K., Saad, W., Bennis, M., Feljan, A. V., & Poor, H. V. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3118346 · Cited by 548
Mobile Edge Intelligence and Computing for the Internet of Vehicles
Zhang, J., & Letaief, K. B. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2947490 · Cited by 537
“In [131], a joint accuracy- and latency-aware (JALAD) execution (a) Perception”
BackgroundEfficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“In sequential distribution, DL layers are assigned to different edge devices [25], [374].”
Background
Metadata
- Authors
- Li, H., Hu, C., Jiang, J., Wang, Z., Wen, Y., & Zhu, W.
- Affiliations
- Tsinghua University (CN)
- Tsinghua Shenzhen International Graduate School (CN)
- Jilin University (CN)
- Nanyang Technological University (SG)
- Venue
- 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS)
- Publisher
- IEEE
- Published
- 2018-12
- Pages
- 671-678
- OpenAlex
- W2949628230
- Cited by
- 160
- References
- 45
- Citations by year
- 2026: 10 · 2025: 21 · 2024: 23 · 2023: 31 · 2022: 29 · 2021: 25 · 2020: 17 · 2019: 4
- FWCI
- 9.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Parallel and Distributed Systems (ICPADS) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- deep neural network deployment, edge-cloud computing, model partitioning, inference latency reduction, accuracy-latency trade-off, edge devices, dynamical decoupling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.