DeepEye: Resource Efficient Local Execution of Multiple Deep Vision Models using Wearable Commodity Hardware
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Wearable devices with built-in cameras present interesting opportunities for users to capture various aspects of their daily life and are potentially also useful in supporting users with low vision in their everyday tasks. However, state-of-the-art image wearables available in the market are limited to capturing images periodically and do not provide any real-time analysis of the data that might be useful for the wearers. In this paper, we present DeepEye - a match-box sized wearable camera that is capable of running multiple cloud-scale deep learn- ing models locally on the device, thereby enabling rich analysis of the captured images in near real-time without offloading them to the cloud. DeepEye is powered by a commodity wearable processor (Snapdragon 410) which ensures its wearable form factor. The software architecture for DeepEye addresses a key limitation with executing multiple deep learning models on constrained hardware, that is their limited runtime memory. We propose a novel inference software pipeline that targets the local execution of multiple deep vision models (specifically, CNNs) by interleaving the execution of computation-heavy convolutional layers with the loading of memory-heavy fully-connected layers. Beyond this core idea, the execution framework incorporates: a memory caching scheme and a selective use of model compression techniques that further minimizes memory bottlenecks. Through a series of experiments, we show that our execution framework outperforms the baseline approaches significantly in terms of inference latency, memory requirements and energy consumption.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534
Deep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,759
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481
DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · Proceedings of the 19th international conference on Architectural support for programming languages and operating systems · doi:10.1145/2541940.2541967 · Cited by 1,297
Age and gender classification using convolutional neural networks
Levi, G., & Hassncer, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301352 · Cited by 1,151
SenseCam: A Retrospective Memory Aid
Hodges, S., Williams, L., Berry, E., Izadi, S., Srinivasan, J., Butler, A., Smyth, G., Kapur, N., & Wood, K. (2006) · Lecture Notes in Computer Science · doi:10.1007/11853565_11 · Cited by 753
Detecting activities of daily living in first-person camera views
Pirsiavash, H., & Ramanan, D. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248010 · Cited by 649
Multi-view Face Detection Using Deep Convolutional Neural Networks
Farfade, S. S., Saberian, M. J., & Li, L.-J. (2015) · Proceedings of the 5th ACM on International Conference on Multimedia Retrieval · doi:10.1145/2671188.2749408 · Cited by 572
Towards wearable cognitive assistance
Ha, K., Chen, Z., Hu, W., Richter, W., Pillai, P., & Satyanarayanan, M. (2014) · Proceedings of the 12th annual international conference on Mobile systems, applications, and services · doi:10.1145/2594368.2594383 · Cited by 476
Understanding my data, myself: supporting self-reflection with ubicomp technologies
Li, I., Dey, A. K., & Forlizzi, J. (2011) · Proceedings of the 13th international conference on Ubiquitous computing · doi:10.1145/2030112.2030166 · Cited by 459
Restructuring of deep neural network acoustic models with singular value decomposition
Xue, J., Li, J., & Gong, Y. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-552 · Cited by 407
PlaNet - Photo Geolocation with Convolutional Neural Networks
Weyand, T., Kostrikov, I., & Philbin, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_3 · Cited by 395
Han, S., Shen, H., Philipose, M., Agarwal, S., Wolman, A., & Krishnamurthy, A. (2016) · Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2906388.2906396 · Cited by 367
Understanding and Predicting Image Memorability at a Large Scale
Khosla, A., Raju, A. S., Torralba, A., & Oliva, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.275 · Cited by 348
Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns
Levi, G., & Hassner, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830587 · Cited by 336 · cited 1×
“EmotionNet [32]: This model aims to recognize the emotion on a person’s face by analyzing the input image.”
BackgroundDo life-logging technologies support memory for the past?: an experimental study using sensecam
Sellen, A. J., Fogg, A., Aitken, M., Hodges, S., Rother, C., & Wood, K. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240636 · Cited by 324 · cited 1×
“…a critical system design issue that will enable important ubiquitous computing and mobile computing applications, many of which are already developed and studied [1, 16, 17, 18, 19, 20, 21], but remain impractical for large-scale deployment because suitable wearable devices are not yet available.”
BackgroundBhattacharya, S., & Lane, N. D. (2016) · Proceedings of the 14th ACM Conference on Embedded Networked Sensor Systems CD-ROM · doi:10.1145/2994551.2994564 · Cited by 236
Self-citationLane, N. D., Bhattacharya, S., Georgiev, P., Forlivesi, C., & Kawsar, F. (2015) · Proceedings of the 2015 International Workshop on Internet of Things towards Applications · doi:10.1145/2820975.2820980 · Cited by 228
Self-citationRedEye: analog ConvNet image sensor architecture for continuous mobile vision
LiKamWa, R., Hou, Y., Gao, J., Polansky, M., & Zhong, L. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001164 · Cited by 189
Energy characterization and optimization of image sensing toward continuous mobile vision
LiKamWa, R., Priyantha, B., Philipose, M., Zhong, L., & Bahl, P. (2013) · Proceeding of the 11th annual international conference on Mobile systems, applications, and services · doi:10.1145/2462456.2464448 · Cited by 175
From smart to deep: Robust activity recognition on smartwatches using deep learning
Bhattacharya, S., & Lane, N. D. (2016) · 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops) · doi:10.1109/percomw.2016.7457169 · Cited by 174
Self-citationEatWell: sharing nutrition-related memories in a low-income community
Grimes, A., Bednar, M., Bolter, J. D., & Grinter, R. E. (2008) · Proceedings of the 2008 ACM conference on Computer supported cooperative work · doi:10.1145/1460563.1460579 · Cited by 154
Kelly, P., Doherty, A., Berry, E., Hodges, S., Batterham, A. M., & Foster, C. (2011) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-8-44 · Cited by 130
From snapshots to social media: the changing picture of domestic photography
(2011) · Choice Reviews Online · doi:10.5860/choice.49-0088 · Cited by 124
Zhang, J., Ma, S., Sameki, M., Sclaroff, S., Betke, M., Lin, Z., Shen, X., Price, B. L., & Měch, R. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299031 · Cited by 103
Predicting daily activities from egocentric images using deep learning
Castro, D., Hickson, S., Bettadapura, V., Thomaz, E., Abowd, G., Christensen, H., & Essa, I. (2015) · Proceedings of the 2015 ACM International Symposium on Wearable Computers - ISWC '15 · doi:10.1145/2802083.2808398 · Cited by 86
Enabling physical analytics in retail stores using smart glasses
Rallapalli, S., Ganesan, A., Chintalapudi, K., Padmanabhan, V. N., & Qiu, L. (2014) · Proceedings of the 20th annual international conference on Mobile computing and networking · doi:10.1145/2639108.2639126 · Cited by 83
Fast Video Classification via Adaptive Cascading of Deep Models
Shen, H., Han, S., Philipose, M., & Krishnamurthy, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.236 · Cited by 66
RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision
LiKamWa, R., Hou, Y., Gao, Y., Polansky, M., & Zhong, L. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.31 · Cited by 61
SR-clustering: Semantic regularized clustering for egocentric photo streams segmentation
Dimiccoli, M., Bolaños, M., Talavera, E., Aghaei, M., Nikolov, S. G., & Radeva, P. I. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.10.005 · Cited by 60
DeepSense: A GPU-based Deep Convolutional Neural Network Framework on Commodity Mobile Devices
Huynh, L. N., Balan, R. K., & Lee, Y. (2016) · Proceedings of the 2016 Workshop on Wearable Systems and Applications · doi:10.1145/2935643.2935650 · Cited by 58
Zhang, J., Ma, S., Sameki, M., Sclaroff, S., Betke, M., Lin, Z., Shen, X., Price, B. L., & Měch, R. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1011-0 · Cited by 37
Lane, N., Bhattacharya, S., Mathur, A., Forlivesi, C., & Kawsar, F. (2016) · Proceedings of the The 8th EAI International Conference on Mobile Computing, Applications and Services · doi:10.4108/eai.30-11-2016.2267463 · Cited by 33
Self-citationDeep-Spying: Spying using Smartwatch and Deep LearningPreprint
Beltramelli, T., & Risi, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.05616 · Cited by 28
Bahl, P., Philipose, M., & Zhong, L. (2012) · Mobile Cloud Computing & Services · doi:10.1145/2307849.2307862 · Cited by 19
(2015) · Cited by 14
Salient Object SubitizingPreprint
Zhang, J., Ma, S., Sameki, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.07525 · Cited by 8
SAPPHIRE: an always-on context-aware computer vision system for portable devices
Venkataramani, S., Bahl, V., Hua, X., Liu, J., Li, J., Philipose, M., Priyantha, N. B., & Shoaib, M. (2015) · Design, Automation, and Test in Europe · doi:10.5555/2755753.2757157 · Cited by 5
Efficient Object Detection via Adaptive Online Selection of Sensor-Array Elements
Philipose, M. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.9147 · Cited by 1
Research works printed but not linked: 4
- 10.5555/1972512 · doi:10.5555/1972512
- N. D. Lane , S. Bhattacharya , C. Forlivesi , P. Georgiev , L. Jiao , L. Qendro ,, and F. Kawsar , " Deepx: A software accelerator for low-power deep learning inference on mobile devices," in IPSN 2016 . N. D. Lane, S. Bhattacharya, C. Forlivesi, P. Georgiev, L. Jiao, L. Qendro,, and F. Kawsar, "Deepx: A software accelerator for low-power deep learning inference on mobile devices," in IPSN 2016.
- 10.1109/iccv.2015.275 · DOI · Paper page
- N. D. Lane , S. Bhattacharya , P. Georgiev , C. Forlivesi , L. Jiao , L. Qendro , and F. Kawsar , " Deepx: A software accelerator for low-power deep learning inference on mobile devices," in Proceedings of the 15th International Conference on Information Processing in Sensor Networks , 2016 . N. D. Lane, S. Bhattacharya, P. Georgiev, C. Forlivesi, L. Jiao, L. Qendro, and F. Kawsar, "Deepx: A software accelerator for low-power deep learning inference on mobile devices," in Proceedings of the 15th International Conference on Information Processing in Sensor Networks, 2016.
Cited by
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455
“DeepEye first segregates DNN layers of all tasks into two pools: convolution layers and fully connected layers.”
InfluentialEdge Assisted Real-time Object Detection for Mobile Augmented Reality
Liu, L., Li, H., & Gruteser, M. (2019) · The 25th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3300061.3300116 · Cited by 552
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“DeepEye [393] is a wearable camera capable of running five DL models on edge for image recognition tasks.”
BackgroundDistributed Artificial Intelligence Empowered by End-Edge-Cloud Computing: A Survey
Duan, S., Wang, D., Ren, J., Lyu, F., Zhang, Y., Wu, H., & Shen, X. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3218527 · Cited by 408
UNPU: An Energy-Efficient Deep Neural Network Accelerator With Fully Variable Weight Bit Precision
Lee, J., Kim, C., Kang, S., Shin, D., Kim, S., & Yoo, H.-J. (2019) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2018.2865489 · Cited by 335
Metadata
- Authors
- Mathur, A., Lane, N. D., Bhattacharya, S., Boran, A., Forlivesi, C., & Kawsar, F.
- Affiliations
- Nokia (United Kingdom) (GB)
- University College London (GB)
- Nokia (Ireland) (IE)
- Nokia (Belgium) (BE)
- Venue
- Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services
- Publisher
- ACM
- Published
- 2017-06-16
- Pages
- 68-81
- OpenAlex
- W2625157458
- Cited by
- 128
- References
- 41
- Citations by year
- 2025: 5 · 2024: 9 · 2023: 14 · 2022: 13 · 2021: 27 · 2020: 23 · 2019: 20 · 2018: 14 · 2017: 3
- FWCI
- 4.80
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Mobile Systems, Applications, and Services (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- wearable cameras, low vision, on-device inference, convolutional neural networks, memory bottleneck, inference latency, memory caching, model compression, real-time image analysis, energy consumption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.