Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge
Open accessAbstract
The computation for today's intelligent personal assistants such as Apple Siri, Google Now, and Microsoft Cortana, is performed in the cloud. This cloud-only approach requires significant amounts of data to be sent to the cloud over the wireless network and puts significant computational pressure on the datacenter. However, as the computational resources in mobile devices become more powerful and energy efficient, questions arise as to whether this cloud-only processing is desirable moving forward, and what are the implications of pushing some or all of this compute to the mobile devices on the edge. In this paper, we examine the status quo approach of cloud-only processing and investigate computation partitioning strategies that effectively leverage both the cycles in the cloud and on the mobile device to achieve low latency, low energy consumption, and high datacenter throughput for this class of intelligent applications. Our study uses 8 intelligent applications spanning computer vision, speech, and natural language domains, all employing state-of-the-art Deep Neural Networks (DNNs) as the core machine learning technique. We find that given the characteristics of DNN algorithms, a fine-grained, layer-level computation partitioning strategy based on the data and computation variations of each layer within a DNN has significant latency and energy advantages over the status quo approach. Using this insight, we design Neurosurgeon, a lightweight scheduler to automatically partition DNN computation between mobile devices and datacenters at the granularity of neural network layers. Neurosurgeon does not require per-application profiling. It adapts to various DNN architectures, hardware platforms, wireless networks, and server load levels, intelligently partitioning computation for best latency or best mobile energy. We evaluate Neurosurgeon on a state-of-the-art mobile development platform and show that it improves end-to-end latency by 3.1X on average and up to 40.7X, reduces mobile energy consumption by 59.5% on average and up to 94.7%, and improves datacenter throughput by 1.5X on average and up to 6.7X.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869 · see citing works
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · see citing works
Caffe: Convolutional Architecture for Fast Feature EmbeddingPreprint
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 4,749 · see citing works
MAUI: making smartphones last longer with code offload
Cuervo, E., Balasubramanian, A., Cho, D.-K., Wolman, A., Saroiu, S., Chandra, R., & Bahl, P. (2010) · Proceedings of the 8th international conference on Mobile systems, applications, and services · doi:10.1145/1814433.1814441 · Cited by 2,281 · see citing works
(2011) · doi:10.1145/1966445.1966473 · Cited by 1,870 · see citing works
cuDNN: Efficient Primitives for Deep LearningPreprint
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 1,394 · see citing works
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2541940.2541967 · Cited by 1,314 · see citing works
A close examination of performance and power characteristics of 4G LTE networks
Huang, J., Qian, F., Gerber, A., Mao, Z. M., Sen, S. B., & Spatscheck, O. (2012) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2307636.2307658 · Cited by 996 · see citing works
Mars, J., Tang, L., Hundt, R., Kevin Skadron, Y., & Soffa, M. L. (2011) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/2155620.2155650 · Cited by 607 · see citing works
Self-citationCOMET: code offload by migrating execution transparently
Gordon, M. S., Jamshidi, D. A., Mahlke, S. A., Mao, Z. M., & Xu, C. (2012) · doi:10.5555/2387880.2387890 · Cited by 370 · see citing works
Ra, M.-R., Sheth, A. N., Mummert, L., Pillai, P. S., Wetherall, D. J., & Govindan, R. (2011) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/1999995.2000000 · Cited by 360 · see citing works
Yang, H., Breslow, A. D., Mars, J., & Tang, L. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/2485922.2485974 · Cited by 346 · see citing works
Self-citationAccelerating Deep Convolutional Neural Networks Using Specialized Hardware
Ovtcharov, K., Ruwase, O., Kim, J.-Y., Fowers, J., Strauß, K., & Chung, E. S. (2015) · Cited by 310 · see citing works
Liu, D., Chen, T., Liu, S., Zhou, J., Zhou, S., Teman, O., Feng, X., Zhou, X., & Chen, Y. (2015) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2694344.2694358 · Cited by 247 · see citing works
Wang, Q., Zhang, X., Zhang, Y., & Yi, Q. (2013) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/2503210.2503219 · Cited by 204 · see citing works
Hauswald, J., Laurenzano, M. A., Zhang, Y., Li, C., Rovinski, A., Khurana, A., Dreslinski, R., Mudge, T., Petrucci, V., Tang, L., et al. (2015) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2694344.2694347 · Cited by 193 · see citing works
Self-citationZhang, Y., Laurenzano, M. A., Mars, J., et al. (2014) · Proceedings of the ... annual International Symposium on Microarchitecture/Proceedings of the annual International Symposium on Microarchitecture · doi:10.1109/micro.2014.53 · Cited by 163 · see citing works
Self-citationHauswald, J., Kang, Y., Laurenzano, M. A., Chen, Q., Li, C., Mudge, T., Dreslinski, R., Mars, J., & Tang, L. (2015) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/2749469.2749472 · Cited by 161 · see citing works
Self-citationTowards energy proportionality for large-scale latency-critical workloads
Lo, D., Cheng, L., Govindaraju, R., Barroso, L. A., & Kozyrakis, C. (2014) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2014.6853237 · Cited by 150 · see citing works
Mars, J., & Tang, L. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/2485922.2485975 · Cited by 148 · see citing works
Self-citationHalpern, M., Zhu, Y., & Reddi, V. J. (2016) · Proceedings - International Symposium on High-Performance Computer Architecture/Proceedings · doi:10.1109/hpca.2016.7446054 · Cited by 148 · see citing works
Accurate and compact large vocabulary speech recognition on mobile devices
Lei, X., Senior, A., Gruenstein, A., et al. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-189 · Cited by 125 · see citing works
BigHouse: A simulation infrastructure for data center systems
Meisner, D., Wu, J., & Wenisch, T. F. (2012) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass.2012.6189204 · Cited by 122 · see citing works
Adrenaline: Pinpointing and reining in tail queries with quick voltage boosting
Hsu, C.-H., Zhang, Y., Laurenzano, M. A., Meisner, D., Wenisch, T. F., Mars, J., Tang, L., & Dreslinski, R. (2015) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2015.7056039 · Cited by 117 · see citing works
Self-citationMobile Network Performance from User Devices: A Longitudinal, Multidimensional Analysis
Nikravesh, A., Choffnes, D., Katz-Bassett, E., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-04918-2_2 · Cited by 103 · see citing works
Octopus-Man: QoS-driven task management for heterogeneous multicores in warehouse-scale computers
Petrucci, V., Laurenzano, M. A., Doherty, J., et al. (2015) · Proceedings - International Symposium on High-Performance Computer Architecture/Proceedings · doi:10.1109/hpca.2015.7056037 · Cited by 97 · see citing works
Protean Code: Achieving Near-Free Online Code Transformations for Warehouse Scale Computers
Laurenzano, M. A., Zhang, Y., Tang, L., et al. (2014) · Proceedings of the ... annual International Symposium on Microarchitecture/Proceedings of the annual International Symposium on Microarchitecture · doi:10.1109/micro.2014.21 · Cited by 60 · see citing works
Self-citationA hybrid approach to offloading mobile image classification
Hauswald, J., Manville, T., Zheng, Q., Dreslinski, R., Chakrabarti, C., & Mudge, T. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6855235 · Cited by 56 · see citing works
Self-citationSkach, M., Arora, M., Hsu, C.-H., et al. (2015) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/2749469.2749474 · Cited by 47 · see citing works
Zhang, Y., Meisner, D., Mars, J., et al. (2016) · ACM/IEEE International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.47 · Cited by 40 · see citing works
Self-citationContinuous shape shifting: Enabling loop co-optimization via near-free dynamic code rewriting
Jain, A., Laurenzano, M. A., Tang, L., & Mars, J. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783726 · Cited by 15 · see citing works
Self-citation(2015) · Cited by 14 · see citing works
Laurenzano, M. A., Zhang, Y., Jiang, C., et al. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001152 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 15
- [1]1. Wearables market to be worth $25 billion by 2019. http://www.ccsinsight.com/press/company- news/2332-wearables-market-to-be-worth-25- billion-by-2019-reveals-ccs-insight. Accessed: 2017-01.
- [19]19. Qian Wang, Xianyi Zhang, Yunquan Zhang, and Qing Yi. Augem: automatically generate high performance dense lin- ear algebra kernels on x86 cpus. In Proceedings of the Inter- national Conference on High Performance Computing, Net- working, Storage and Analysis, page 25. ACM, 2013.
- [29]29. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Bur- get, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In Proc. ASRU, 2011.
- [30]30. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural lan- guage processing (almost) from scratch. The Journal of Ma- chine Learning Research, 2011.
- [33]33. Mark Slee, Aditya Agarwal, and Marc Kwiatkowski. Thrift: Scalable cross-language services implementation. Facebook White Paper, 5(8), 2007.
- [36]36. Moo-Ryong Ra, Anmol Sheth, Lily Mummert, Padmanabhan Pillai, David Wetherall, and Ramesh Govindan. Odessa: en- abling interactive perception applications on mobile devices. In Proceedings of the 9th international conference on Mobile systems, applications, and services, pages 43-56. ACM, 2011.
- [41]41. Jason Mars, Lingjia Tang, Robert Hundt, Kevin Skadron, and Mary Lou Soffa. Bubble-up: Increasing utilization in modern warehouse scale computers via sensible co-locations. In In- ternational Symposium on Microarchitecture (MICRO), 2011.
- [43]43. Jason Mars and Lingjia Tang. Whare-map: Heterogeneity in "homogeneous" warehouse-scale computers. In International Symposium on Computer Architecture (ISCA), 2013.
- [45]45. Johann Hauswald, Michael A. Laurenzano, Yunqi Zhang, Cheng Li, Austin Rovinski, Arjun Khurana, Ronald G. Dres- linski, Trevor Mudge, Vinicius Petrucci, Lingjia Tang, and Jason Mars. Sirius: An open end-to-end voice and vision personal assistant and its implications for future warehouse scale computers. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2015.
- [46]46. Hailong Yang, Alex Breslow, Jason Mars, and Lingjia Tang. Bubble-flux: Precise online qos management for increased utilization in warehouse scale computers. In International Symposium on Computer Architecture (ISCA), 2013.
- [49]49. Quan Chen, Hailong Yang, Jason Mars, and Lingjia Tang. Baymax: Qos awareness and increased utilization for non- preemptive accelerators in warehouse scale computers. In ACM SIGPLAN Notices, volume 51, pages 681-696. ACM, 2016.
- [52]52. Michael A. Laurenzano, Yunqi Zhang, Jiang Chen, Lingjia Tang, and Jason Mars. Powerchop: Identifying and manag- ing non-critical units in hybrid processor architectures. In Proceedings of the 43rd International Symposium on Com- puter Architecture, ISCA '16, pages 140-152, Piscataway, NJ, USA, 2016. IEEE Press.
- [53]53. Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengy- ong Wu, Yunji Chen, and Olivier Temam. Diannao: A small- footprint high-throughput accelerator for ubiquitous machine- learning. In Proceedings of the 19th international conference on Architectural support for programming languages and op- erating systems, pages 269-284. ACM, 2014.
- [54]54. Daofu Liu, Tianshi Chen, Shaoli Liu, Jinhong Zhou, Shengyuan Zhou, Olivier Teman, Xiaobing Feng, Xuehai Zhou, and Yunji Chen. Pudiannao: A polyvalent machine learning accelerator. In Proceedings of the Twentieth Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems, pages 369-381. ACM, 2015.
- [57]57. Xin Lei, Andrew Senior, Alexander Gruenstein, and Jeffrey Sorensen. Accurate and compact large vocabulary speech recognition on mobile devices. In INTERSPEECH, pages 662-665, 2013.
Cited by
Sorted by citations
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455 · see citing works
Deep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540 · see citing works
Model inversion attacks against collaborative inference
He, Z., Zhang, T., & Lee, R. B. (2019) · Proceedings of the 35th Annual Computer Security Applications Conference · doi:10.1145/3359789.3359824 · Cited by 313 · see citing works
Extend Cloud to Edge with KubeEdge
Xiong, Y., Sun, Y., Xing, L., & Huang, Y. (2018) · 2018 IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec.2018.00048 · Cited by 206 · see citing works
On-edge multi-task transfer learning: model and practice with data-driven task allocation
Chen, Q., Zheng, Z., Hu, C., Wang, D., & Liu, F. (2020) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2019.2962435 · Cited by 88 · see citing works
Li, G., Liu, L., Wang, X., Dong, X., Zhao, P., & Feng, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01418-6_40 · Cited by 80 · see citing works
- Authors
- Kang, Y., Hauswald, J., Gao, C., Rovinski, A., Mudge, T., Mars, J., & Tang, L.
- Affiliations
- University of Michigan (US)
- Venue
- ACM SIGPLAN Notices
- Published
- 2017-04-04
- Volume
- 52
- Issue
- 4
- Pages
- 615-629
- ISSN
- 0362-1340 (ISSN-L)
- 1558-1160
- OpenAlex
- W4235435541
- Cited by
- 185
- References
- 37
- Citations by year
- 2026: 32 · 2025: 25 · 2024: 19 · 2023: 25 · 2022: 25 · 2021: 20 · 2020: 8 · 2019: 28 · 2018: 2 · 2017: 1
- FWCI
- 7.11
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGPLAN Notices (publishedVersion)
- Funders
- National Science Foundation (IIS-VEC- 1539011,CCF-SHF-1302682,CNS-CSR-1321047)
- Intel (IIS-VEC-1539011)
- ARM
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Keywords
- computation partitioning, edge computing, cloud computing, deep neural network, virtual assistants, mobile devices, latency, energy consumption, neurosurgeons
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
