DeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications
The DOI matches its Crossref record; 44 of the 46 references it cites were found in a registry or an index.
Abstract
The rapid emergence of head-mounted devices such as the Microsoft Holo-lens enables a wide variety of continuous vision applications. Such applications often adopt deep-learning algorithms such as CNN and RNN to extract rich contextual information from the first-person-view video streams. Despite the high accuracy, use of deep learning algorithms in mobile devices raises critical challenges, i.e., high processing latency and power consumption. In this paper, we propose DeepMon, a mobile deep learning inference system to run a variety of deep learning inferences purely on a mobile device in a fast and energy-efficient manner. For this, we designed a suite of optimization techniques to efficiently offload convolutional layers to mobile GPUs and accelerate the processing; note that the convolutional layers are the common performance bottleneck of many deep learning models. Our experimental results show that DeepMon can classify an image over the VGG-VeryDeep-16 deep learning model in 644ms on Samsung Galaxy S7, taking an important step towards continuous vision without imposing any privacy concerns nor networking cost.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“Also, ResNet-152 with 152 layers shows 94.3% top-5 accuracy.”
BackgroundInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 3×
“, VGG-Verydeep-16 [46] and YOLO [43]) to identify their performance bottlenecks.”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 11×
“It takes 6345ms for basic-CPU to process an image using AlexNet, which is 45.6 times slower than DeepMon.”
MethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“There are other models such as Faster-RCNN [22] for object detection and Long-term Recurrent Convolutional Networks (LRCN [19]) for activity recognition – LRCN is the combination of CNN and Long Short Term Memory (LSTM [27]).”
MethodYou Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · cited 6×
“Note that FastYOLO [43], a lightweight version of YOLO shows the lower mAP of 52.”
BackgroundMethodInfluentialImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 3×
“We used the VGG-VeryDeep-16 model trained with the ILSVRC2012 train dataset [44] and YOLO trained with the Pascal VOC 2007 train dataset [8].”
MethodHistograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · cited 1×
“For YOLO, we used the mean average precision (mAP), which is a standard metric to evaluate the YOLO’s accuracy regarding both object recognition and localization [21].”
MethodAdvances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · cited 8×
“We additionally validated the latency of convolutional layers with another commonly-used library, ViennaCL (on Caffe).”
MethodInfluentialThe Pascal Visual Object Classes Challenge: A Retrospective
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0733-5 · Cited by 7,139 · cited 1×
“For the accuracy evaluation, we used the ILSVRC2012 [44] validation dataset for VGG-VeryDeep-16 and the Pascal VOC 2007 test dataset [20] for YOLO, and calculated the average accuracy over each test dataset.”
MethodParkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479 · cited 2×
“VGG-Face [42], is based on VGG-VeryDeep-16, and performs face recognition.”
MethodLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 1×
“There are other models such as Faster-RCNN [22] for object detection and Long-term Recurrent Convolutional Networks (LRCN [19]) for activity recognition – LRCN is the combination of CNN and Long Short Term Memory (LSTM [27]).”
MethodUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · cited 1×
“For the latency evaluation, we used two test datasets: (i) the UCF101 dataset [48] comprising 13,421 short videos (less than a minute long) created for activity recognition and (ii) LENA dataset [47] consists of 200 first-person-view videos captured from Google Glasses, and report the average latency across all proces…”
MethodReturn of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
MatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444 · cited 2×
“The model loader then loads the generated model on DeepMon– it allocates adequate memory spaces to lay out input data for efficient convolution computation and structures the processors for all the layers composing the model (See Section 6.2) DeepMon currently supports the models from three different deep learning fra…”
MethodInfluentialMAUI: making smartphones last longer with code offload
Cuervo, E., Balasubramanian, A., Cho, D.-K., Wolman, A., Saroiu, S., Chandra, R., & Bahl, P. (2010) · Proceedings of the 8th international conference on Mobile systems, applications, and services · doi:10.1145/1814433.1814441 · Cited by 2,281 · cited 2×
“We do plan to extend our solution to support cyber foraging (e.g. MAUI [16] and Chroma [10, 11]), where local and cloud resources are used in a dynamic fashion.”
BackgroundASIFT: A New Framework for Fully Affine Invariant Image Comparison
Morel, J.-M., & Yu, G. (2009) · SIAM Journal on Imaging Sciences · doi:10.1137/080732730 · Cited by 1,468 · cited 3×
“Although SIFT-based algorithm provides the lower output difference ratio (3.875”
MethodResultInfluentialExploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1404.0736 · Cited by 1,414 · cited 1×
“Restructuring DNN models has been widely studied to reduce the size of the model and accelerate the inference speed [18, 23, 29, 32, 34, 45].”
MethodSpeeding up Convolutional Neural Networks with Low Rank Expansions
Jaderberg, M., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.88 · Cited by 1,164
(2011) · Cited by 713
Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets
Sainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638949 · Cited by 593 · cited 1×
“Restructuring DNN models has been widely studied to reduce the size of the model and accelerate the inference speed [18, 23, 29, 32, 34, 45].”
MethodA literature survey of low‐rank tensor approximation techniques
Grasedyck, L., Kressner, D., & Tobler, C. (2013) · GAMM-Mitteilungen · doi:10.1002/gamm.201310004 · Cited by 577 · cited 1×
“Restructuring DNN models has been widely studied to reduce the size of the model and accelerate the inference speed [18, 23, 29, 32, 34, 45].”
Method(2006) · Cited by 490
Towards wearable cognitive assistance
Ha, K., Chen, Z., Hu, W., Richter, W., Pillai, P., & Satyanarayanan, M. (2014) · Proceedings of the 12th annual international conference on Mobile systems, applications, and services · doi:10.1145/2594368.2594383 · Cited by 476 · cited 1×
“Gabriel [24] uses cloudlets to support cognitive assistance applications while LiKamWa et al.”
MethodHan, S., Shen, H., Philipose, M., Agarwal, S., Wolman, A., & Krishnamurthy, A. (2016) · Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2906388.2906396 · Cited by 367 · cited 7×
“One way to overcome this limitation is to use cloud resources to run the required DNNs [25].”
BackgroundMethodInfluentialGlimpse: Continuous, Real-Time Object Recognition on Mobile Devices
Chen, T. Y.-H., Ravindranath, L., Deng, S., Bahl, P., & Balakrishnan, H. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809711 · Cited by 358
DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning
Lane, N. D., Georgiev, P., & Qendro, L. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2804262 · Cited by 310 · cited 3×
“took important first steps towards the real-time execution of DNN and CNN on mobile devices [35, 36].”
BackgroundMethodSpeeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I. V., & Lempitsky, V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6553 · Cited by 283 · cited 1×
“Prior research have provided a few different methods (such as the tucker decomposition [32] and CP decomposition [37]) to decompose a convolutional layer into three smaller convolutional layers so that the total computation of the decomposed layers is less than that of the original layer.”
MethodFast Training of Convolutional Networks through FFTs
Mathieu, M., Henaff, M., & LeCun, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.5851 · Cited by 273 · cited 12×
“In the machine learning communities, there has been work to reduce the training time of CNNs and DNNs [40].”
BackgroundInfluentialTactics-based remote execution for mobile computing
Balan, R. K., Satyanarayanan, M., Park, S. Y., & Okoshi, T. (2003) · Proceedings of the 1st international conference on Mobile systems, applications and services · doi:10.1145/1066116.1066125 · Cited by 248 · cited 2×
“We do plan to extend our solution to support cyber foraging (e.g. MAUI [16] and Chroma [10, 11]), where local and cloud resources are used in a dynamic fashion.”
BackgroundSelf-citationBhattacharya, S., & Lane, N. D. (2016) · Proceedings of the 14th ACM Conference on Embedded Networked Sensor Systems CD-ROM · doi:10.1145/2994551.2994564 · Cited by 236
Boger, J., Hoey, J., Poupart, P., Boutilier, C., Fernie, G., & Mihailidis, A. (2006) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2006.864480 · Cited by 180 · cited 1×
“These range from identifying road signs in real time to provide directions [15], to identifying people in the environment to give guidance to individuals suffering from dementia [12].”
BackgroundSimplifying cyber foraging for mobile devices
Balan, R. K., Gergle, D., Satyanarayanan, M., & Herbsleb, J. (2007) · Proceedings of the 5th international conference on Mobile systems, applications and services · doi:10.1145/1247660.1247692 · Cited by 175 · cited 2×
“We do plan to extend our solution to support cyber foraging (e.g. MAUI [16] and Chroma [10, 11]), where local and cloud resources are used in a dynamic fashion.”
BackgroundSelf-citationEnergy characterization and optimization of image sensing toward continuous mobile vision
LiKamWa, R., Priyantha, B., Philipose, M., Zhong, L., & Bahl, P. (2013) · Proceeding of the 11th annual international conference on Mobile systems, applications, and services · doi:10.1145/2462456.2464448 · Cited by 175 · cited 2×
“In this paper, we focused on reducing the power consumption of executing deep learning pipeline on a mobile device and rely optimising the power consumption of the video camera (to capture and store continuous video feeds) to prior work [38].”
MethodTenth international workshop on frontiers in handwriting recognition
Lorette, G., Horst Bunke, A., & Schomaker, L. (2006) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 175
Kim, B., Lee, M., Kim, J., Kim, J., & Lee, J. (2016) · Advanced science and technology letters · doi:10.14257/astl.2016.140.36 · Cited by 149
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
Kim, Y., Park, E., Yoo, S., Choi, T., Lu, Y., & Shin, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06530 · Cited by 111 · cited 8×
“Tucker-2 decomposition is more appropriate to be used with our caching technique due to its unique characteristic – two of the decomposed layers have the filters with the size of [1 x 1].”
MethodInfluentialDo Deep Convolutional Nets Really Need to be Deep and Convolutional?
Urban, G., Geras, K. J., Kahou, S. E., Aslan, Ö., Wang, S., Caruana, R., Mohamed, A., Philipose, M., & Richardson, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.05691 · Cited by 95 · cited 1×
“This increase in accuracy has been validated by recent results [49] (although there are a few special cases where a shallow network is equally accurate).”
ResultDeepSense: A GPU-based Deep Convolutional Neural Network Framework on Commodity Mobile Devices
Huynh, L. N., Balan, R. K., & Lee, Y. (2016) · Proceedings of the 2016 Workshop on Wearable Systems and Applications · doi:10.1145/2935643.2935650 · Cited by 58 · cited 1×
“DeepSense [28] presented early evidence that using a GPU could help improve the latency of DNN computations – DeepMon extends that work by providing many more optimisations, a full implementation, and extensive evaluation.”
Self-citationStarfish: Efficient Concurrency Support for Computer Vision Applications
LiKamWa, R., & Zhong, L. (2015) · Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2742647.2742663 · Cited by 55 · cited 2×
“presented optimization techniques for image sensors to enable continuous mobile vision [38] and Starfish [39] to support concurrent execution of multiple vision applications.”
Activity Recognition in Egocentric Life-Logging Videos
Song, S., Chandrasekhar, V., Cheung, N. M., Narayan, S., Li, L., & Lim, J. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16634-6_33 · Cited by 32 · cited 1×
“For the latency evaluation, we used two test datasets: (i) the UCF101 dataset [48] comprising 13,421 short videos (less than a minute long) created for activity recognition and (ii) LENA dataset [47] consists of 200 first-person-view videos captured from Google Glasses, and report the average latency across all proces…”
MethodSparsifying Deep Learning Layers for Constrained Resource Inference on Wearables
Lane, N., & Bhattacharya, S. (2016) · UCL Discovery (University College London) · Cited by 12
Research works printed but not linked: 2
- Kim , Y.-D. , Park , E. , Yoo , S. , Choi , T. , Yang , L. , and Shin , D . Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530 , 2015 . Kim, Y.-D., Park, E., Yoo, S., Choi, T., Yang, L., and Shin, D. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530, 2015. · arXiv
- 10.5555/2959355.2959378 · doi:10.5555/2959355.2959378
Cited by
Deep Learning With Edge Computing: A Review
Chen, J., & Ran, X. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2921977 · Cited by 1,540
“DeepMon [78] combines quantization with caching of results from intermediate layers on GPUs.”
BackgroundMethodConvergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498
Edge Assisted Real-time Object Detection for Mobile Augmented Reality
Liu, L., Li, H., & Gruteser, M. (2019) · The 25th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3300061.3300116 · Cited by 552
DeepDecision: A Mobile Deep Learning Framework for Edge Video Analytics
Ran, X., Chen, H., Zhu, X., Liu, Z., & Chen, J. (2018) · IEEE INFOCOM 2018 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2018.8485905 · Cited by 450
“Hence, any system that performs offloading of real-time scenes must take this delay into account when measuring the accuracy; however, previous works generally compute the accuracy relative to the original time of frame capture [8], [7].”
BackgroundMethodInfluentialEfficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“DeepMon [162] uses a combination of quantization and caching to speed up the execution by reducing redundant computations of video frames.”
Method
Metadata
- Authors
- Huynh, L. N., Lee, Y., & Balan, R. K.
- Affiliations
- Singapore Management University (SG)
- Venue
- Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services
- Publisher
- ACM
- Published
- 2017-06-16
- Pages
- 82-95
- OpenAlex
- W2626129225
- Cited by
- 329
- References
- 46
- Citations by year
- 2026: 6 · 2025: 24 · 2024: 23 · 2023: 24 · 2022: 38 · 2021: 51 · 2020: 72 · 2019: 50 · 2018: 35 · 2017: 6
- FWCI
- 12.83
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Mobile Systems, Applications, and Services (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Funders
- National Research Foundation Singapore
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Keywords
- head-mounted devices, energy-efficient inference, processing delay
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.