Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Open accessThe DOI matches its Crossref record; 408 of the 491 references it cites were found in a registry or an index.
Abstract
Successful integration of deep neural networks (DNNs) or deep learning (DL) has resulted in breakthroughs in many areas. However, deploying these highly accurate models for data-driven, learned, automatic, and practical machine learning (ML) solutions to end-user applications remains challenging. DL algorithms are often computationally expensive, power-hungry, and require large memory to process complex and iterative operations of millions of parameters. Hence, training and inference of DL models are typically performed on high-performance computing (HPC) clusters in the cloud. Data transmission to the cloud results in high latency, round-trip delay, security and privacy concerns, and the inability of real-time decisions. Thus, processing on edge devices can significantly reduce cloud transmission cost. Edge devices are end devices closest to the user, such as mobile phones, cyber–physical systems (CPSs), wearables, the Internet of Things (IoT), embedded and autonomous systems, and intelligent sensors. These devices have limited memory, computing resources, and power-handling capability. Therefore, optimization techniques at both the hardware and software levels have been developed to handle the DL deployment efficiently on the edge. Understanding the existing research, challenges, and opportunities is fundamental to leveraging the next generation of edge devices with artificial intelligence (AI) capability. Mainly, four research directions have been pursued for efficient DL inference on edge devices: 1) novel DL architecture and algorithm design; 2) optimization of existing DL methods; 3) development of algorithm–hardware codesign; and 4) efficient accelerator design for DL deployment. This article focuses on surveying each of the four research directions, providing a comprehensive review of the state-of-the-art tools and techniques for efficient edge inference.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These data cannot be managed by relying only on the limited functional services provided by cloud service centers (Ge et al. 2023;Shuvo et al. 2022).”
From Survey: federated learning data security and privacy-preserving in edge-Internet of Things (2024) · cites this paper as Shuvo et al. 2022
Citations
Cites
Items 201–400 of 410
When Edge Meets Learning: Adaptive Control for Resource-Constrained Distributed Machine Learning
Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., & Chan, K. (2018) · IEEE INFOCOM 2018 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2018.8486403 · Cited by 209 · cited 1×
“For instance, training and inference of ML models that use gradient calculation for optimization (such as SVM, K-means, and LR) are demonstrated in [28].”
MethodInteractive machine learning: letting users build classifiers
WARE, M., FRANK, E., HOLMES, G., HALL, M., & WITTEN, I. H. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2001.0499 · Cited by 207 · cited 1×
“Interactive ML (iML) involves users, providing information to the learning process and monitoring the output in subsequent iterations [356], [357].”
BackgroundDianNao family: energy-efficient hardware accelerators for machine learning
Chen, Y., Chen, T., Xu, Z., Sun, N., & Temam, O. (2016) · Communications of the ACM · doi:10.1145/2996864 · Cited by 205
Deep Reinforcement LearningPreprint
Li, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06339 · Cited by 204
HeteroCL: A Multi-Paradigm Programming Infrastructure for Software-Defined Reconfigurable Computing
Lai, Y., Chi, Y., Hu, Y., Wang, J., Yu, C. H., Zhou, Y., Cong, J., & Zhang, Z. (2019) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3289602.3293910 · Cited by 201 · cited 2×
“Frameworks such as DNNWeaver [414], DNNBuilder [415], T2S-Tensor [416], and HeteroCL [417] have been developed recently for FPGA, ASIC, and other DL hardware accelerators for efficient EI.”
MethodCapra, M., Bussolino, B., Marchisio, A., Shafique, M., Masera, G., & Martina, M. (2020) · Future Internet · doi:10.3390/fi12070113 · Cited by 195 · cited 1×
“Reviews on hardware realization of convolutional NNs (CNNs) [8] in general and on the application-specific integrated circuit (ASIC) [9] have also been presented.”
BackgroundWang, T., Zhu, J.-Y., Torralba, A., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1811.10959 · Cited by 194 · cited 1×
“The knowledge from a large dataset is transferred to a smaller dataset that significantly reduces computational loads of training DL models [193], [194].”
BackgroundMa, X., Guo, F.-M., Niu, W., Lin, X., Tang, J., Ma, K., Ren, B., & Wang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5954 · Cited by 193 · cited 1×
“A few prominent examples include PCONV [261], PatDNN [262], and CoCoPIE [263], which considers simultaneous design exploration compression and compiler.”
BackgroundGuo, X., Bayat, F. M., Bavandpour, M., Klachko, M., Mahmoodi, M. R., Prezioso, M., Likharev, K. K., & Strukov, D. B. (2017) · 2017 IEEE International Electron Devices Meeting (IEDM) · doi:10.1109/iedm.2017.8268341 · Cited by 192 · cited 2×
“The traditional SRAM [338], DRAM [339], and Flash [340] memories also can be modified to realize the PIM implementation of DNN algorithms.”
MethodUCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition
Hegde, K., Yu, J., Agrawal, R., Yan, M., Pellauer, M., & Fletcher, C. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00062 · Cited by 191 · cited 1×
“Unique weight CNN (UCNN) accelerator [251] reduces the memory size considering a generalization of the sparsity.”
BackgroundTraining Deep Neural Networks with 8-bit Floating Point Numbers
Wang, N., Choi, J., Brand, D., Chen, C., & Gopalakrishnan, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08011 · Cited by 189
ScaleDeep: A Scalable Compute Architecture for Learning and Evaluating Deep Networks
Venkataramani, S., Ranjan, A., Banerjee, S., Das, D., Avancha, S., Jagannathan, A., Durg, A., Nagaraj, D., Kaul, B., Dubey, P., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080244 · Cited by 188 · cited 1×
“Though accelerators such as ScaleDeep [419] and HyPar [420] are proposed for training on edge, these architectures do not consider the sparsity, variable precision of gradients, parameters, and activations.”
MethodLi, C. (2019) · Reinforcement Learning for Cyber-Physical Systems · doi:10.1201/9781351006620-6 · Cited by 188
Yao, S., Zhao, Y., Zhang, A., Su, L., & Abdelzaher, T. (2017) · Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems · doi:10.1145/3131672.3131675 · Cited by 188
DeepCache: Principled Cache for Mobile Deep Vision
Xu, M., Zhu, M., Liu, Y., Lin, F. X., & Liu, X. (2018) · Proceedings of the 24th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3241539.3241563 · Cited by 185 · cited 1×
“CNNCache [24] reuses the results from similar image regions, thus minimizing computations and the burden on device resources.”
BackgroundEfficient and Effective Sparse LSTM on FPGA with Bank-Balanced Sparsity
Cao, S., Zhang, C., Yao, Z., Xiao, W., Nie, L., Zhan, D., Liu, Y., Wu, M., & Zhang, L. (2019) · Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3289602.3293898 · Cited by 178 · cited 2×
“Unstructured pruning results in irregular computation and memory access that limit parallelism for hardware implementations [96], [97].”
BackgroundMethodBinary convolutional neural network on RRAM
Tang, T., Xia, L., Li, B., Wang, Y., & Yang, H. (2017) · 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/aspdac.2017.7858419 · Cited by 176 · cited 1×
“Therefore, BNN can be realized easily by utilizing the memory architectures and logic operations [341].”
MethodFrom smart to deep: Robust activity recognition on smartwatches using deep learning
Bhattacharya, S., & Lane, N. D. (2016) · 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops) · doi:10.1109/percomw.2016.7457169 · Cited by 174 · cited 1×
“RBMs are used in [388] for HAR applications on smartwatches.”
MethodSnaPEA: Predictive Early Activation for Reducing Computation in Deep Convolutional Neural Networks
Akhlaghi, V., Yazdanbakhsh, A., Samadi, K., Gupta, R. K., & Esmaeilzadeh, H. (2018) · 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2018.00061 · Cited by 173 · cited 2×
“The number of operations can be lessened by observing the sign of accumulated activation in the PE [240].”
BackgroundDeep Reinforcement Learning: Frontiers of Artificial Intelligence
Sewak, M. (2019) · Springer eBooks · doi:10.1007/978-981-13-8285-7 · Cited by 173
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1712.00559 · Cited by 171 · cited 1×
“Progressive NAS (PNAS) is reported in [66] for CNN architecture search.”
BackgroundLearning to Prune Filters in Convolutional Neural Networks
Huang, Q., Zhou, K., You, S., & Neumann, U. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00083 · Cited by 170 · cited 2×
“A structured pruning technique presented in [100] for CNN removes certain filters to increase the efficiency of the network without significantly affecting the accuracy.”
BackgroundMethodDeep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
Wang, E., Davis, J. J., Zhao, R., Ng, H.-C., Niu, X., Luk, W., Cheung, P. Y. K., & Constantinides, G. A. (2020) · ACM Computing Surveys · doi:10.1145/3309551 · Cited by 166
A 0.3–2.6 TOPS/W precision-scalable processor for real-time large-scale ConvNets
Moons, B., & Verhelst, M. (2016) · 2016 IEEE Symposium on VLSI Circuits (VLSI-Circuits) · doi:10.1109/vlsic.2016.7573525 · Cited by 161 · cited 1×
“AlexNet can be quantized using a variable precision of 4 to 9 bits with less than 1% accuracy degradation [119].”
MethodfpgaConvNet: Mapping regular and irregular convolutional neural networks on FPGAs
Venieris, S. I., & Bouganis, C.-S. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2844093 · Cited by 161 · cited 2×
“Due to the increased demand for EI, small and high-density FPGAs are now being considered [11], [315], [316], [317].”
BackgroundMethodA Hybrid Deep Learning Architecture for Privacy-Preserving Mobile Analytics
Osia, S. A., Shahin Shamsabadi, A., Sajadmanesh, S., Taheri, A., Katevas, K., Rabiee, H. R., Lane, N. D., & Haddadi, H. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2967734 · Cited by 161 · cited 1×
“Researchers proposed different techniques for appropriately performing the DL model splitting, such as differential private mechanism [365], Arden (privAte infeRence framework based on DNNs) [366], and hybrid edge-cloud framework [367].”
MethodLee, J., Lee, J., Han, D., Lee, J., Park, G., & Yoo, H.-J. (2019) · 2019 IEEE International Solid-State Circuits Conference - (ISSCC) · doi:10.1109/isscc.2019.8662302 · Cited by 161 · cited 1×
“The advantage of such sparsification has been examined in CNN models to be about 40% and 50% MAC operation elimination during training and inference, respectively [232].”
BackgroundJALAD: Joint Accuracy-And Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution
Li, H., Hu, C., Jiang, J., Wang, Z., Wen, Y., & Zhu, W. (2018) · 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/padsw.2018.8645013 · Cited by 160 · cited 1×
“In sequential distribution, DL layers are assigned to different edge devices [25], [374].”
BackgroundApproxANN: An Approximate Computing Framework for Artificial Neural Network
Zhang, Q., Wang, T., Tian, Y., Yuan, F., & Xu, Q. (2015) · Design, Automation & Test in Europe Conference & Exhibition (DATE), 2015 · doi:10.7873/date.2015.0618 · Cited by 158
YOLO Nano: a Highly Compact You Only Look Once Convolutional Neural Network for Object Detection
Wong, A., Famuori, M., Shafiee, M. J., Li, F., Chwyl, B., & Chung, J. (2019) · 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) · doi:10.1109/emc2-nips53020.2019.00013 · Cited by 155
Zhu, M., Zhang, T., Gu, Z., & Xie, Y. (2019) · Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1145/3352460.3358269 · Cited by 153 · cited 2×
“Unstructured pruning results in irregular computation and memory access that limit parallelism for hardware implementations [96], [97].”
BackgroundMethodPipeDream: Fast and Efficient Pipeline Parallel DNN Training
Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., & Gibbons, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03377 · Cited by 153 · cited 1×
“For instance, PipeDream is model parallelism through pipelined training presented in [368].”
MethodScalpel: Customizing DNN Pruning to the Underlying Hardware Parallelism
Yu, J., Lukefahr, A., Palframan, D., Dasika, G., Das, R., & Mahlke, S. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080215 · Cited by 152
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., & He, Y. (2022) · arXiv · doi:10.48550/arxiv.2206.01861 · Cited by 151 · cited 1×
“Compression techniques, such as ZeroQuant [295], are designed, and many existing compression techniques are built-in in the DeepSpeed library for rapid compression, reduced model size, and faster inference.”
MethodZeRO-Offload: Democratizing Billion-Scale Model Training
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06840 · Cited by 151
Cachier: Edge-Caching for Recognition Applications
Drolia, U., Guo, K., Tan, J., Gandhi, R., & Narasimhan, P. (2017) · 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2017.94 · Cited by 148 · cited 2×
“A 3 × improvement of latency is possible using Cachier [21].”
BackgroundMethodShiDianNao: shifting vision processing closer to the sensor
Du, Z., Fasthuber, R., Chen, T., Ienne, P., Li, L., Luo, T., Feng, X., Chen, Y., & Temam, O. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/2872887.2750389 · Cited by 148
Energy Harvesting in Implantable and Wearable Medical Devices for Enduring Precision Healthcare
Shuvo, M. M. H., Titirsha, T., Amin, N., & ISLAM, S. K. (2022) · Energies · doi:10.3390/en15207495 · Cited by 147 · cited 1×
“Although alternate power sources are emerging for edge devices through energy harvesting [252], reducing the power consumption of inference is crucial.”
BackgroundSelf-citationKung, H. T., McDanel, B., & Zhang, S. Q. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304028 · Cited by 146 · cited 2×
“The number of address indexes is reduced in the block sparsity, where a single address index is used for a group of weights [238], [239].”
BackgroundLogNet: Energy-efficient neural networks using logarithmic computation
Lee, E. H., Miyashita, D., Chai, E., Murmann, B., & Wong, S. S. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953288 · Cited by 145 · cited 1×
“8% accuracy degradation for 4-bit precision linear quantization [150].”
BackgroundIONN: Incremental Offloading of Neural Network Computations from Mobile Devices to Edge Servers
Jeong, H.-J., Lee, H.-J., Shin, C. H., & Moon, S.-M. (2018) · Proceedings of the ACM Symposium on Cloud Computing · doi:10.1145/3267809.3267828 · Cited by 143 · cited 1×
“In [26], instead of uploading the entire model to the cloud, the layers are incrementally uploaded for collaborative inference, which significantly improves the energy cost and performance.”
MethodI-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141 · cited 1×
“An integer-only quantization of vision transformers for computer vision tasks is reported in [156].”
BackgroundNeural Networks with Few Multiplications
Lin, Z., Courbariaux, M., Memisevic, R., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03009 · Cited by 141
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2022) · arXiv · Cited by 140
DeltaRNN: A Power-efficient Recurrent Neural Network Accelerator
Gao, C., Neil, D., Ceolini, E., Liu, S.-C., & Delbruck, T. (2018) · Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3174243.3174261 · Cited by 133
SCOPE: A Stochastic Computing Engine for DRAM-Based In-Situ Accelerator
Li, S., Glova, A. O., Hu, X., Gu, P., Niu, D., Malladi, K. T., Zheng, H., Brennan, B. A., & Xie, Y. (2018) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2018.00062 · Cited by 131 · cited 1×
“The traditional SRAM [338], DRAM [339], and Flash [340] memories also can be modified to realize the PIM implementation of DNN algorithms.”
MethodBig/little deep neural network for ultra low power inference
Park, E., Kim, D., Kim, S., Kim, Y.-D., Kim, G., Yoon, S., & Yoo, S. (2015) · 2015 International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS) · doi:10.1109/codesisss.2015.7331375 · Cited by 130 · cited 1×
“Two DL models of different sizes (big and little) can be used for classification during inference [210].”
MethodMathur, A., Lane, N. D., Bhattacharya, S., Boran, A., Forlivesi, C., & Kawsar, F. (2017) · Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3081333.3081359 · Cited by 128 · cited 1×
“DeepEye [393] is a wearable camera capable of running five DL models on edge for image recognition tasks.”
BackgroundExploring Sparsity in Recurrent Neural Networks
Narang, S., Elsen, E., Diamos, G., & Sengupta, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.05119 · Cited by 126
Flexible Dataset Distillation: Learn Labels Instead of ImagesPreprint
Bohdal, O., Yang, Y., & Hospedales, T. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08572 · Cited by 126
Rocket Launching: A Universal and Efficient Framework for Training Well-Performing Light Net
Zhou, G., Fan, Y., Cui, R., Bian, W., Zhu, X., & Gai, K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11601 · Cited by 121 · cited 1×
“Rocket launching [187] is a parallel learning scheme that trains both the teacher and student models simultaneously.”
BackgroundPermDNN: Efficient Compressed DNN Architecture with Permuted Diagonal Matrices
Deng, C., Liao, S., Xie, Y., Parhi, K. K., Qian, X., & Yuan, B. (2018) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2018.00024 · Cited by 120 · cited 1×
“The number of address indexes is reduced in the block sparsity, where a single address index is used for a group of weights [238], [239].”
BackgroundInvestigating gated recurrent networks for speech synthesis
Wu, Z., & King, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472657 · Cited by 118
Yuan, Z., Yue, J., Yang, H., Wang, Z., Li, J., Yang, Y., Guo, Q., Li, X., Chang, M.-F., Yang, H., et al. (2018) · 2018 IEEE Symposium on VLSI Circuits · doi:10.1109/vlsic.2018.8502404 · Cited by 117 · cited 2×
“Within a 2.4-mm 2 chip area, it can achieve an energy efficiency of 21.55 TOPS/W for convolutional layers, and it can run pruned ResNet-50 inference within a power consumption of 348 mW. STICKER [310] is an ASIC for efficient inference of DL models containing convolution and dense layers.”
MethodInfluentialDistributed Inference with Deep Learning Models across Heterogeneous Edge Devices
Hu, C., & Li, B. (2022) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom48880.2022.9796896 · Cited by 112 · cited 1×
“To facilitate distributed inference for DL models having a directed acyclic graph (DAG) structure, EdgeFlow is reported in [380].”
MethodDave, S., Baghdadi, R., Nowatzki, T., Avancha, S., Shrivastava, A., & Li, B. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3098483 · Cited by 111 · cited 1×
“Handling sparse and irregular tensors resulted in different DL models, and techniques in hardware are summarized in [15].”
MethodCompression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
Kim, Y., Park, E., Yoo, S., Choi, T., Lu, Y., & Shin, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06530 · Cited by 111
ZeNA: Zero-Aware Neural Network Accelerator
Kim, D., Ahn, J., & Yoo, S. (2018) · IEEE Design & Test · doi:10.1109/mdat.2017.2741463 · Cited by 109 · cited 1×
“Recent developments of inference accelerators, such as ZENA [235], SNAP [236], and EyerissV2 [237], leverage dynamic sparsity.”
MethodGANAX: A Unified MIMD-SIMD Acceleration for Generative Adversarial Networks
Yazdanbakhsh, A., Falahati, H., Wolfe, P. J., Esmaeilzadeh, H., & Samadi, K. (2018) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2018.00060 · Cited by 109 · cited 1×
“Transposed convolutions in the GANs insert zeros by upscaling resulting in 60% MAC elimination [222].”
BackgroundAccelerating CNN Inference on ASICs: A Survey
Moolchandani, D., Kumar, A., & Sarangi, S. R. (2020) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2020.101887 · Cited by 108 · cited 2×
“Since 2012, DL has experienced a 3000 × and 10 × increase in computational need and memory requirements, respectively [9].”
BackgroundHyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array
Song, L., Mao, J., Zhuo, Y., Qian, X., Li, H., & Chen, Y. (2019) · 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2019.00027 · Cited by 108 · cited 1×
“Though accelerators such as ScaleDeep [419] and HyPar [420] are proposed for training on edge, these architectures do not consider the sparsity, variable precision of gradients, parameters, and activations.”
MethodEscher: A CNN Accelerator with Flexible Buffering to Minimize Off-Chip Transfer
Shen, Y., Ferdman, M., & Milder, P. (2017) · 2017 IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm.2017.47 · Cited by 106 · cited 1×
“The map batching technique balances the heavy transfer of parameters in dense layers by kernel reuse eliminating the transfer to an external memory [111].”
MethodMoonshine: Distilling with Cheap Convolutions
Crowley, E. J., Gray, G., & Storkey, A. (2017) · arXiv · doi:10.48550/arxiv.1711.02613 · Cited by 106
Architectural Complexity Measures of Recurrent Neural Networks
Zhang, S., Wu, Y., Che, T., Lin, Z., Memisevic, R., Salakhutdinov, R., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08210 · Cited by 105
Design Flow of Accelerating Hybrid Extremely Low Bit-Width Neural Network in Embedded FPGA
Wang, J., Lou, Q., Zhang, X., Zhu, C., Lin, Y., & Chen, D. (2018) · 2018 28th International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2018.00035 · Cited by 104 · cited 1×
“Often, low bit width is used for middle layer weights, and large bit-width quantization is used for weights in the first and last layers [128].”
MethodSystolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference
Liu, Z.-G., Whatmough, P. N., & Mattina, M. (2020) · IEEE Computer Architecture Letters · doi:10.1109/lca.2020.2979965 · Cited by 104 · cited 1×
“Such architectures allow off-line structured arrangements of sparse tensors [234].”
BackgroundZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., & He, Y. (2022) · neural information processing systems · doi:10.52202/068431-1970 · Cited by 104
Zeng, X., Cao, K., & Zhang, M. (2017) · Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3081333.3081336 · Cited by 103 · cited 1×
“MobileDeepPill [395] is an image recognition system to identify pills for patients from pill images.”
BackgroundSynetgy: Algorithm-hardware Co-design for ConvNet Accelerators on Embedded FPGAs
Yang, Y., Huang, Q., Wu, B., Zhang, T., Ma, L., Gambardella, G., Blott, M., Lavagno, L., Vissers, K., Wawrzynek, J., et al. (2019) · Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3289602.3293902 · Cited by 102 · cited 1×
“For FPGA implementations, spatial convolutions are replaced by shift operations and depth convolutions followed by 1 × 1 convolutions in DiracDeltaNet [164].”
MethodAdaptive deep learning model selection on embedded systems
Taylor, B., Marco, V. S., Wolff, W., Elkhatib, Y., & Wang, Z. (2018) · ACM SIGPLAN Notices · doi:10.1145/3299710.3211336 · Cited by 102
Improved Knowledge Distillation via Teacher Assistant
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., & Ghasemzadeh, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5963 · Cited by 101 · cited 1×
“Various applications with impressive results have been developed for KD, such as mutual learning [190], teacher assistant [191], and self-learning [192].”
BackgroundElsen, E., Dukhan, M., Gale, T., & Simonyan, K. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01464 · Cited by 101 · cited 1×
“For compact network architectures, such as MobileNet-v2 and EfficientNet-B0, pruning can introduce 50%–93% sparsity resulting in 2.5 × –4.2 × MAC elimination [223].”
MethodInfluentialPrediction Based Execution on Deep Neural Networks
Song, M., Zhao, J., Hu, Y., Zhang, J., & Li, T. (2018) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2018.00068 · Cited by 100 · cited 1×
“The computations for only the important neurons are realized in [241].”
BackgroundA Scalable Multi- TeraOPS Deep Learning Processor Core for AI Trainina and Inference
Fleischer, B., Shukla, S., Ziegler, M., Silberman, J., Oh, J., Srinivasan, V., Choi, J., Mueller, S., Agrawal, A., Babinsky, T., et al. (2018) · 2018 IEEE Symposium on VLSI Circuits · doi:10.1109/vlsic.2018.8502276 · Cited by 100 · cited 1×
“Such coarse-grained architectures are more power-efficient for DL inference execution compared to the CPU and GPU cores [250], [318], [319].”
ResultGate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
You, Z., Yan, K., Ye, J., Ma, M., & Wang, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.08174 · Cited by 100
Factorization tricks for LSTM networks
Kuchaiev, O., & Ginsburg, B. (2017) · arXiv · doi:10.48550/arxiv.1703.10722 · Cited by 99 · cited 1×
“The weight matrix of LSTM can be factorized into matrices, and results can be concatenated for faster convergence and parameter reduction [90].”
BackgroundShuvo, M. M. H., Ahmed, N., Nouduri, K., & Palaniappan, K. (2020) · IEEE Applied Imagery Pattern Recognition Workshop · doi:10.1109/aipr50011.2020.9425332 · Cited by 98 · cited 1×
“HAR finds significant applications in wearable sensors for fitness tracking, activity monitoring, patient monitoring, fall detection, elder care, and so on [387].”
BackgroundSelf-citationCNNdroid: GPU-Accelerated Execution of Trained Deep Convolutional Neural Networks on Android
Latifi Oskouei, S. S., Golestani, H., Hashemi, M., & Ghiasi, S. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2973801 · Cited by 95 · cited 1×
“For instance, CNNdroid [276] is a GPU-based library for DCNN inference on the android smartphone that can result in 60 × speedup and 130 × energy reduction.”
MethodDeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices
Li, D., Wang, X., & Kong, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11876 · Cited by 94 · cited 1×
“For example, DeepRebirth [186] merges convolution layers with consecutive weightless layers, such as pooling and normalization, together to get a compressed student model.”
Wang, P., Ji, Y., Hong, C., Lyu, Y., Wang, D., & Xie, Y. (2018) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3195970.3196116 · Cited by 94
A 1.06-to-5.09 TOPS/W reconfigurable hybrid-neural-network processor for deep learning applications
Yin, S., Ouyang, P., Tang, S., Tu, F., Li, X., Liu, L., & Wei, S. (2017) · 2017 Symposium on VLSI Circuits · doi:10.23919/vlsic.2017.8008534 · Cited by 93 · cited 1×
“In [321], a reconfigurable ASIC implemented in the 65-nm process is presented targeting hybrid NNs.”
MethodYOLObile: Real-Time Object Detection on Mobile Devices via Compression-Compilation Co-Design
Cai, Y., Li, H., Yuan, G., Niu, W., Li, Y., Tang, X., Ren, B., & Wang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16179 · Cited by 93 · cited 1×
“D activity detection using C3D, R(2 + 1)D, and S3D networks achieving 19 frames/s and 6.8 ms per frame performance without accuracy degradation, respectively, in smartphones [275].”
MethodInfluentialMultiplier-less Artificial Neurons Exploiting Error Resiliency for Energy-Efficient Neural Computing
Sarwar, S. S., Venkataramani, S., Raghunathan, A., & Roy, K. (2016) · Design, Automation, and Test in Europe · doi:10.3850/9783981537079_0848 · Cited by 93
FoggyCache: Cross-Device Approximate Computation Reuse
Guo, P., Hu, B., Li, R., & Hu, W. (2018) · Proceedings of the 24th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3241539.3241557 · Cited by 91 · cited 1×
“FoggyCache [23] uses caching to reduce the redundant computation and achieves 3 × latency improvement and 10 × reduction of energy cost.”
MethodLoom: Exploiting Weight and Activation Precisions to Accelerate Convolutional Neural Networks
Sharify, S., Lascorz, A. D., Siu, K., Judd, P., & Moshovos, A. (2018) · 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC) · doi:10.1109/dac.2018.8465915 · Cited by 91
Runtime configurable deep neural networks for energy-accuracy trade-off
Tann, H., Hashemi, S., Bahar, R. I., & Reda, S. (2016) · International Conference on Hardware/Software Codesign and System Synthesis · doi:10.1145/2968456.2968458 · Cited by 89
Ando, K., Ueyoshi, K., Orimo, K., Yonekawa, H., Sato, S., Nakahara, H., Ikebe, M., Asai, T., Takamaeda-Yamazaki, S., Kuroda, T., et al. (2017) · 2017 Symposium on VLSI Circuits · doi:10.23919/vlsic.2017.8008533 · Cited by 87 · cited 1×
“For instance, binary values are used for weights in YodaNN [130], and both the weights and activations adopt binary values in BRein [131].”
MethodDeepCham: Collaborative Edge-Mediated Adaptive Deep Learning for Mobile Object Recognition
Li, D., Salonidis, T., Desai, N. V., & Chuah, M. C. (2016) · 2016 IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec.2016.38 · Cited by 84 · cited 1×
“DeepCham [350] consists of one edge server connected to multiple mobile devices.”
BackgroundA scalable sparse matrix-vector multiplication kernel for energy-efficient sparse-blas on FPGAs
Dorrance, R., Ren, F., & Marković, D. (2014) · Proceedings of the 2014 ACM/SIGDA international symposium on Field-programmable gate arrays · doi:10.1145/2554688.2554785 · Cited by 83 · cited 1×
“The CSC provides better memory efficiency compared to the CSR format [247].”
MethodAlternating Multi-bit Quantization for Recurrent Neural Networks
Xu, C., Yao, J., Lin, Z., Ou, W., Cao, Y., Wang, Z., & Zha, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00150 · Cited by 82 · cited 1×
“For instance, improved accuracy has been achieved in [153] using the binary quantization to RNN.”
MethodSC-DCNN: Highly-Scalable Deep Convolutional Neural Network using Stochastic Computing
Ren, A., Li, Z., Ding, C., Qiu, Q., Wang, Y., Li, J., Qian, X., & Yuan, B. (2017) · ACM SIGPLAN Notices · doi:10.1145/3093336.3037746 · Cited by 81
Deep Model Compression: Distilling Knowledge from Noisy Teachers
Sau, B. B., & Balasubramanian, V. N. (2016) · Research Archive of Indian Institute of Technology Hyderabad (Indian Institute of Technology Hyderabad) · doi:10.48550/arxiv.1610.09650 · Cited by 81
DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices
Hou, X., Guan, Y., Han, T., & Zhang, N. (2022) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps53621.2022.00110 · Cited by 80 · cited 1×
“DistrEdge [379] is an adaptive distributed EI framework for CNN models considering the heterogeneity of edge devices, network conditions, and nonlinear computations.”
MethodAccelerating Neural Network Inference on FPGA-Based Platforms—A Survey
Wu, R., Guo, X., Du, J., & Li, J. (2021) · Electronics · doi:10.3390/electronics10091025 · Cited by 79 · cited 1×
“There are available reviews of different DL techniques [5], DNNs on edge [2], [16], DL compression techniques for hardware acceleration [10], DL inference on FPGA-based platforms [17], edge intelligence [18], and edge computing [19].”
BackgroundXpulpNN: Accelerating Quantized Neural Networks on RISC-V Processors Through ISA Extensions
Garofalo, A., Tagliavini, G., Conti, F., Rossi, D., & Benini, L. (2020) · 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE) · doi:10.23919/date48585.2020.9116529 · Cited by 78 · cited 1×
“In [333], an extension of the RISC-V ISA has been proposed that supports subbyte quantization.”
MethodLow Latency Deep Learning Inference Model for Distributed Intelligent IoT Edge Clusters
Naveen, S., Kounte, M. R., & Ahmed, M. R. (2021) · IEEE Access · doi:10.1109/access.2021.3131396 · Cited by 77 · cited 1×
“In another similar work, five IoT and a gateway device are employed that uses weight pruning and model partitioning using FTP [377].”
MethodDarnet: a deep learning solution for distracted driving detection
Streiffer, C., Raghavendra, R., Benson, T., & Srivatsa, M. (2017) · Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference: Industrial Track · doi:10.1145/3154448.3154452 · Cited by 77 · cited 1×
“In Dar-Net [396], driving patterns are analyzed using DL to identify distracted driving.”
MethodLin, J., Zhu, Z., Wang, Y., & Xie, Y. (2019) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3287624.3287715 · Cited by 77 · cited 1×
“The weight matrix can also be decomposed into blocks to cluster the zero columns for coarse-grained pruning [343], [344].”
BackgroundSmartwatch Based Activity Recognition Using Active Learning
Shahmohammadi, F., Hosseini, A., King, C. E., & Sarrafzadeh, M. (2017) · 2017 IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE) · doi:10.1109/chase.2017.115 · Cited by 76 · cited 1×
“In [358], iML is exploited in edge training for human activity recognition (HAR) and demonstrated that a few training iterations achieve acceptable performance.”
BackgroundMobiFace: A Lightweight Deep Learning Face Recognition on Mobile Devices
Duong, C. N., Quach, K. G., Jalata, I., Le, N., & Luu, K. (2019) · 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS) · doi:10.1109/btas46853.2019.9185981 · Cited by 76 · cited 1×
“MobiFace [386] is an improvement on Mobile-FaceNet using residual bottleneck [72] with an expansion layer and fast downsampling.”
Wang, B., Ma, F., Ge, L., Ma, H., Wang, H., & Mohamed, M. A. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3018831 · Cited by 75 · cited 1×
“Lightweight DL models can also be used for transmission line fault detection [409] with 74.5% accuracy.”
MethodInfluentialMoving Deep Learning to the Edge
Véstias, M. P., Duarte, R. P., de Sousa, J. T., & Neto, H. C. (2020) · Algorithms · doi:10.3390/a13050125 · Cited by 73 · cited 1×
“There are available reviews of different DL techniques [5], DNNs on edge [2], [16], DL compression techniques for hardware acceleration [10], DL inference on FPGA-based platforms [17], edge intelligence [18], and edge computing [19].”
BackgroundA Robust Deep Learning Approach for Position-Independent Smartphone-Based Human Activity Recognition
Almaslukh, B., Artoli, A. M., & Al-Muhtadi, J. (2018) · Sensors · doi:10.3390/s18113726 · Cited by 71 · cited 1×
“Statistical features of CNN are used in [389], which extended, in [390], adding the position features for HAR.”
MethodSeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization
Cao, S., Ma, L., Xiao, W., Zhang, C., Liu, Y., Zhang, L., Nie, L., & Yang, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01147 · Cited by 71 · cited 1×
“For instance, activations are 40% and 70% sparse in CNN and FC layers, respectively, [220] due to ReLU activations.”
BackgroundDeep Learning Inference on Embedded Devices: Fixed-Point vs Posit
Langroudi, S. H. F., Pandit, T., & Kudithipudi, D. (2018) · Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2) · doi:10.1109/emc2.2018.00012 · Cited by 69 · cited 1×
“For example, in [122], CNN is compressed by using posit numbers to read and write weights from/to memory.”
MethodBonawitz, K., Kairouz, P., McMahan, B., & Ramage, D. (2021) · Queue · doi:10.1145/3494834.3500240 · Cited by 69 · cited 1×
“After local training, the model updates are aggregated in a central cloud server and usually averaged [361].”
MethodThe unreasonable effectiveness of the forget gate
van der Westhuizen, J., & Lasenby, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.04849 · Cited by 68 · cited 1×
“The units of S-LSTM [85] and JANET [86] consist of only one forget gate.”
BackgroundWider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence LearningPreprint
He, Z., Gao, S., Xiao, L., Liu, D., He, H., & Barber, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01577 · Cited by 68
Georgiev, P., Bhattacharya, S., Lane, N. D., & Mascolo, C. (2017) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3131895 · Cited by 67 · cited 1×
“The accuracy can be further enhanced by training shared layers and multitask learning [398].”
BackgroundAutomated Pruning for Deep Neural Network Compression
Manessi, F., Rozza, A., Bianco, S., Napoletano, P., & Schettini, R. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8546129 · Cited by 67 · cited 1×
“To overcome this limitation, differentiable pruning [104] can be used.”
MethodAddNet: Deep Neural Networks Using FPGA-Optimized Multipliers
Faraone, J., Kumm, M., Hardieck, M., Zipf, P., Liu, X., Boland, D., & Leong, P. H. W. (2020) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2019.2939429 · Cited by 66
Stripes: Bit-Serial Deep Neural Network Computing
Judd, P., Albericio, J., & Moshovos, A. (2016) · IEEE Computer Architecture Letters · doi:10.1109/lca.2016.2597140 · Cited by 66 · cited 1×
“Stripes [123] is an implementation of variable bit width with bit-serial computation.”
BackgroundDNPU: An Energy-Efficient Deep-Learning Processor with Heterogeneous Multi-Core Architecture
Shin, D., Lee, J., Lee, J., Lee, J., & Yoo, H. (2018) · IEEE Micro · doi:10.1109/mm.2018.053631145 · Cited by 65 · cited 1×
“The DNPU is another CGRA processor [322] that supports CNN and RNN processing.”
MethodT2S-Tensor: Productively Generating High-Performance Spatial Hardware for Dense Tensor Computations
Srivastava, N., Rong, H., Barua, P., Feng, G., Cao, H., Zhang, Z., Albonesi, D. H., Sarkar, V., Chen, W., Petersen, P. M., et al. (2019) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm.2019.00033 · Cited by 64 · cited 2×
“Frameworks such as DNNWeaver [414], DNNBuilder [415], T2S-Tensor [416], and HeteroCL [417] have been developed recently for FPGA, ASIC, and other DL hardware accelerators for efficient EI.”
MethodBlood Glucose Prediction in Type 1 Diabetes Using Deep Learning on the Edge
Zhu, T., Kuang, L., Li, K., Zeng, J., Herrero, P., & Georgiou, P. (2021) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas51556.2021.9401083 · Cited by 63
Fully Distributed Deep Learning Inference on Resource-Constrained Edge Devices
Stahl, R., Zhao, Z., Mueller-Gritschneder, D., Gerstlauer, A., & Schlichtmann, U. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-27562-4_6 · Cited by 60 · cited 1×
“A communication-aware distributed inference across six edge devices is presented in [372] for YOLOv2 inference.”
MethodEnergy Efficient Neural Computing: A Study of Cross-Layer Approximations
Sarwar, S. S., Srinivasan, G., Han, B., Wijesinghe, P., Jaiswal, A., Panda, P., Raghunathan, A., & Roy, K. (2018) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2018.2835809 · Cited by 59 · cited 1×
“At the hardware level, an approximation could be in architectural optimization [167], approximate data storage [168], and reading writing speed modifications [169].”
BackgroundDeepSense: A GPU-based Deep Convolutional Neural Network Framework on Commodity Mobile Devices
Huynh, L. N., Balan, R. K., & Lee, Y. (2016) · Proceedings of the 2016 Workshop on Wearable Systems and Applications · doi:10.1145/2935643.2935650 · Cited by 58 · cited 1×
“DeepSense [281] is a GPU-based CNN inference framework for mobile devices with improved latency.”
BackgroundCrossbar-Aware Neural Network Pruning
Liang, L., Deng, L., Zeng, Y., Hu, X., Ji, Y., Ma, X., Li, G., & Xie, Y. (2018) · IEEE Access · doi:10.1109/access.2018.2874823 · Cited by 52 · cited 1×
“The weight matrix can also be decomposed into blocks to cluster the zero columns for coarse-grained pruning [343], [344].”
BackgroundZimmer, B., Venkatesan, R., Shao, Y. S., Clemons, J., Fojtik, M., Jiang, N., Keller, B., Klinefelter, A., Pinckney, N., Raina, P., et al. (2019) · 2019 Symposium on VLSI Circuits · doi:10.23919/vlsic.2019.8778056 · Cited by 52 · cited 1×
“Tianjic [324] and Simba [325] are two spatial architectures that utilized the concept of distributed weights [326] and weight exchange among PEs [319].”
BackgroundApproximation-aware Multi-Level Cells STT-RAM cache architecture
Sampaio, F., Shafique, M., Zatt, B., Bampi, S., & Henkel, J. (2015) · International Conference on Compilers, Architecture, and Synthesis for Embedded Systems (CASES) · doi:10.1109/cases.2015.7324548 · Cited by 52 · cited 1×
“At the hardware level, an approximation could be in architectural optimization [167], approximate data storage [168], and reading writing speed modifications [169].”
BackgroundSqueezeFlow: A Sparse CNN Accelerator Exploiting Concise Convolution Rules
Li, J., Jiang, S., Gong, S., Wu, J., Yan, J., Yan, G., & Li, X. (2019) · IEEE Transactions on Computers · doi:10.1109/tc.2019.2924215 · Cited by 51 · cited 1×
“SqueezeFlow [249] used the RLC for weights supporting both the sparse and dense models.”
MethodRSTensorFlow: GPU Enabled TensorFlow for Deep Learning on Commodity Android Devices
Alzantot, M., Wang, Y., Ren, Z., & Srivastava, M. B. (2017) · Proceedings of the 1st International Workshop on Deep Learning for Mobile Systems and Applications · doi:10.1145/3089801.3089805 · Cited by 50 · cited 2×
“Parallel computing of CNNs and RNNs has demonstrated faster response in CPU, GPU, and DSP [259].”
MethodMesh-TensorFlow: Deep Learning for Supercomputers
Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.02084 · Cited by 50
MobiRNN: Efficient Recurrent Neural Network Execution on Mobile GPU
Cao, Q., Balasubramanian, N., & Balasubramanian, A. (2017) · Proceedings of the 1st International Workshop on Deep Learning for Mobile Systems and Applications · doi:10.1145/3089801.3089804 · Cited by 49 · cited 1×
“However, directly employing the CUDA for mobile GPU may deteriorate the system performance [278].”
MethodEnabling deep learning at the IoT edge
Lai, L., & Suda, N. (2018) · Proceedings of the International Conference on Computer-Aided Design · doi:10.1145/3240765.3243473 · Cited by 49
ShapeShifter: Enabling Fine-Grain Data Width Adaptation in Deep Learning
Lascorz, A. D., Sharify, S., Edo, I., Stuart, D. M., Awad, O. M., Judd, P., Mahmoud, M., Nikolić, M., Siu, K., Poulos, Z., et al. (2019) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3352460.3358295 · Cited by 49 · cited 1×
“Weights and activations can also be grouped to use different bit widths to encode values of different groups [129].”
BackgroundHitNet: Hybrid Ternary Recurrent Neural Network
Wang, P., Xie, X., Deng, L., Li, G., Wang, D., & Xie, Y. (2018) · neural information processing systems · Cited by 49
A Dynamic Deep Neural Network Design for Efficient Workload Allocation in Edge Computing
Lo, C., Su, Y.-Y., Lee, C.-Y., & Chang, S.-C. (2017) · 2017 IEEE International Conference on Computer Design (ICCD) · doi:10.1109/iccd.2017.49 · Cited by 48 · cited 1×
“An authentic operation (AO) unit is added to BranchyNet in [216] to determine whether the input is to be passed to the edge server or cloud based on the confidence score.”
BackgroundZhang, J.-F., Lee, C.-E., Liu, C., Shao, Y. S., Keckler, S. W., & Zhang, Z. (2019) · 2019 Symposium on VLSI Circuits · doi:10.23919/vlsic.2019.8778193 · Cited by 47 · cited 3×
“It is capable of processing 34.4 frames/s with an energy efficiency of 1.32 TOPS/W. SNAP [309] is another ASIC designed in a 16-nm process that adopted unstructured sparsity for optimization.”
MethodInfluentialMASR: A Modular Accelerator for Sparse RNNs
Gupta, U., Reagen, B., Pentecost, L., Donato, M., Tambe, T., Rush, A. M., Wei, G.-Y., & Brooks, D. (2019) · 2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact.2019.00009 · Cited by 47 · cited 1×
“For instance, refactoring the batch normalization for RNN in MASR [230] achieves about 60% activation sparsity.”
MethodHan, D., Lee, J., Lee, J., & Yoo, H. (2019) · Symposium on VLSI Circuits · doi:10.23919/vlsic.2019.8778006 · Cited by 43 · cited 1×
“An ASIC for object tracking using pipelined direct feedback alignment (PDFA) is presented in [308].”
MethodAxTrain: Hardware-Oriented Neural Network Training for Approximate Inference
He, X. J., Ke, L., Lü, W., Yan, G., & Zhang, X. (2018) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1145/3218603.3218643 · Cited by 42 · cited 1×
“AxTrain [173] also leverages AC that improves the inference on hardware by finding the error-tolerant parameters.”
BackgroundActions at the Edge: Jointly Optimizing the Resources in Multi-Access Edge Computing
Deng, Y., Chen, X., Zhu, G., Fang, Y. M., Chen, Z., & Deng, X. (2022) · IEEE Wireless Communications · doi:10.1109/mwc.006.2100699 · Cited by 42
Liu, S., Du, J., Nan, K., Zhou, Z., Liu, H., Wang, Z., & Lin, Y. (2020) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2020.2999956 · Cited by 41 · cited 1×
“To meet the resource constraints of edge devices, Adadeep [161] presents an automatic selection of compression techniques and filters.”
MethodEfficient Winograd Convolution via Integer ArithmeticPreprint
Meng, L., & Brothers, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.01965 · Cited by 41 · cited 1×
“13× [80], respectively, using the Winograd algorithm.”
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation
Ge, T., Chen, S., & Wei, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.741 · Cited by 41 · cited 2×
“For example, EdgeFormer [381] is a transformer-based encoder–decoder architecture designed for seq2seq generation considering the memory and computation constraints of edge devices.”
MethodStructured Pruning for Efficient ConvNets via Incremental Regularization
Wang, H., Zhang, Q., Wang, Y., Yu, L., & Hu, H. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852463 · Cited by 39 · cited 1×
“2) Regularization-Based: These methods use a regularization part in the loss function equation to provide the required sparsity level [107].”
MethodStream Processing Dual-Track CGRA for Object Inference
Fan, X., Wu, D., Cao, W., Luk, W. W., & Wang, L. (2018) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2018.2797600 · Cited by 38 · cited 1×
“The SDT-CGRA [323] architecture includes up to three MAC instructions.”
Method1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed
Li, C., Awan, A. A., Tang, H., Rajbhandari, S., & He, Y. (2022) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc56025.2022.00044 · Cited by 38 · cited 1×
“In addition, 1-bit Adam [292], 0/1 Adam [293], and 1-bit LAMB [294] optimizers reduce the communication resource demand in DeepSpeed.”
BackgroundYao, S., Zhao, Y., Zhang, A., Su, L., & Abdelzaher, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01215 · Cited by 37 · cited 1×
“In [158], pruning methods are implemented for popular DL models targeting IoT and mobile devices.”
MethodTask Scheduling with Optimized Transmission Time in Collaborative Cloud-Edge Learning
Huang, Y., Zhu, Y., Fan, X., Ma, X., Wang, F., Liu, J., Wang, Z., & Cui, Y. (2018) · 2018 27th International Conference on Computer Communication and Networks (ICCCN) · doi:10.1109/icccn.2018.8487352 · Cited by 36 · cited 1×
“Scheduling schemes for edge training are presented in [352], where the edge devices upload a training profile requesting cloud access.”
BackgroundA communication efficient distributed learning framework for smart environments
Valerio, L., Passarella, A., & Conti, M. (2017) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2017.07.014 · Cited by 35 · cited 2×
“Peer-to-peer distributed training is another infrastructure in which edge devices are given equal priority [353].”
BackgroundResultA robust convolutional neural network for online smartphone-based human activity recognition
Almaslukh, B., Al-Muhtadi, J. F., & Artoli, A. M. (2018) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-169699 · Cited by 34 · cited 1×
“Statistical features of CNN are used in [389], which extended, in [390], adding the position features for HAR.”
MethodA Simplified Fully Quantized Transformer for End-to-end Speech RecognitionPreprint
Bie, A., Venkitesh, B., Monteiro, J., Haidar, M. A., & Rezagholizadeh, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03604 · Cited by 34 · cited 2×
“For instance, exploiting 8-bit fixed point precision for weights and activations, SpeechTransformer [155] achieves a 4× reduced number of parameters while maintaining the performance of a full-precision model.”
Method(2021) · The MIT Press eBooks · doi:10.7551/mitpress/12832.003.0015 · Cited by 34
Lane, N., Bhattacharya, S., Mathur, A., Forlivesi, C., & Kawsar, F. (2016) · Proceedings of the The 8th EAI International Conference on Mobile Computing, Applications and Services · doi:10.4108/eai.30-11-2016.2267463 · Cited by 33 · cited 1×
“The DeepX toolkit (DXTK) [285] is a generalization of DeepX techniques for other DL models and layers.”
Method(2017) · doi:10.1145/3132211.3134456 · Cited by 32 · cited 1×
“In Precog [22], the cached data are stored both at the edge server and mobile devices that achieve a 5 × speedup.”
MethodEmgAuth: An EMG-based Smartphone Unlocking System Using Siamese Network
Fan, B., Liu, X., Su, X. J., Hui, P., & Niu, J. (2020) · IEEE International Conference on Pervasive Computing and Communications · doi:10.1109/percom45495.2020.9127387 · Cited by 32 · cited 1×
“For instance, face recognition is being actively used in door unlocking, mobile banking [383], device unlocking [384], and so on.”
BackgroundFujii, T., Toi, T., Tanaka, T., et al. (2018) · IEEE Symposium on VLSI Circuits (VLSI-Circuits) · doi:10.1109/vlsic.2018.8502438 · Cited by 31 · cited 1×
“A dynamically reconfigurable processor (DRP) is a CGRA core to accelerate inference on embedded devices [327].”
MethodReconfigurable Convolutional Kernels for Neural Networks on FPGAs
Hardieck, M., Kumm, M., Möller, K., & Zipf, P. (2019) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3289602.3293905 · Cited by 31 · cited 1×
“Thus, the CNN layer can exploit data reuse through a reconfigurable convolutional kernel [83].”
MethodUniWiG: Unified Winograd-GEMM Architecture for Accelerating CNN on FPGAs
Kala, S., Mathew, J., Jose, B. R., & Nalesh, S. (2019) · International Conference on VLSI Design · doi:10.1109/vlsid.2019.00055 · Cited by 29 · cited 1×
“A unified architecture incorporating the Winograd and general matrix multiplication (GEMM) named UniWig is presented in [82].”
MethodCoCoPIE: enabling real-time AI on off-the-shelf mobile devices via compression-compilation co-design
Guan, H., Liu, S., Ma, X., Niu, W., Ren, B., Shen, X., Wang, Y., & Zhao, P. (2021) · Communications of the ACM · doi:10.1145/3418297 · Cited by 29 · cited 2×
“A few prominent examples include PCONV [261], PatDNN [262], and CoCoPIE [263], which considers simultaneous design exploration compression and compiler.”
BackgroundMethodApproximate memory compression for energy-efficiency
Ranjan, A., Raha, A., Raghunathan, V., & Raghunathan, A. (2017) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped.2017.8009173 · Cited by 28 · cited 1×
“At the hardware level, an approximation could be in architectural optimization [167], approximate data storage [168], and reading writing speed modifications [169].”
BackgroundEnergy efficient parallel computing on the PULP platform with support for OpenMP
Rossi, D., Loi, I., Conti, F., et al. (2014) · IEEE Convention of Electrical and Electronics Engineers in Israel · doi:10.1109/eeei.2014.7005803 · Cited by 27 · cited 2×
“The parallel ultralow-power (PULP-NN) library [332] can help to deploy DL on GAP8 processors.”
BackgroundAre Very Deep Neural Networks Feasible on Mobile Devices
Rallapalli, S., Qiu, H., Bency, A. J., Karthikeyan, S., Govindan, R., Manjunath, B. S., & Urgaonkar, R. (2016) · Cited by 27
(2018) · Cited by 27
Cappuccino: Efficient CNN Inference Software Synthesis for Mobile System-on-Chips
Motamedi, M., Fong, D. D., & Ghiasi, S. (2018) · IEEE Embedded Systems Letters · doi:10.1109/les.2018.2815954 · Cited by 25 · cited 2×
“Cappuccino takes in the model definition, model file, and dataset as input and applies three levels of parallelization on kernel, filter, and outputs.”
MethodFast Convolutional Neural Networks in Low Density FPGAs Using Zero-Skipping and Weight Pruning
Véstias, M. P., Duarte, R. P., de Sousa, J. T., & Neto, H. C. (2019) · Electronics · doi:10.3390/electronics8111321 · Cited by 25
An Automatic and Efficient BERT Pruning for Edge AI Systems
Huang, S., Liu, N., Liang, Y., Peng, H., Li, H., Xu, D., Xie, M., & Ding, C. (2022) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed54688.2022.9806197 · Cited by 25
DeepSearch: A Fast Image Search Framework for Mobile Devices
Wang, P., Hu, Q., Fang, Z., Zhao, C., & Cheng, J. (2017) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3152127 · Cited by 24
Self-citationANTNets: Mobile Convolutional Neural Networks for Resource Efficient Image ClassificationPreprint
Xiong, Y., Kim, H. J., & Hedau, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03775 · Cited by 24 · cited 1×
“ANTNet [77] is an image classification CNN that also utilized the group convolution concept.”
MethodHARNet: Towards On-Device Incremental Learning using Deep Ensembles on Constrained Devices
Sundaramoorthy, P., Gudur, G. K., Moorthy, M. R., Bhandari, R. N., & Vijayaraghavan, V. (2018) · International Workshop on Embedded and Mobile Deep Learning (EMDL) · doi:10.1145/3212725.3212728 · Cited by 23 · cited 1×
“Due to the heterogeneous sensor types and distinct user behavior, incremental learning [391] provides secondary training for the pretrained model to learn new features from incoming data.”
BackgroundStaged Inference using Conditional Deep Learning for energy efficient real-time smart diagnosis
Parsa, M., Panda, P., Sen, S., & Roy, K. (2017) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2017.8036767 · Cited by 23 · cited 1×
“A medical diagnostic tool based on hierarchical inference is developed using wearable sensors in [219].”
MethodSquishedNets: Squishing SqueezeNet further for edge device scenarios via deep evolutionary synthesisPreprint
Shafiee, M. J., Li, F., Chwyl, B., & Wong, A. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07459 · Cited by 23
AxDNN: towards the cross-layer design of approximate DNNs
Fan, Y., Wu, X., Dong, J., & Qi, Z. (2019) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3287624.3287627 · Cited by 23 · cited 1×
“AxDNN [171] is the cross-layer approximation of a DNN incorporating activation pruning, approximate multiplier, and voltage scaling to improve energy efficiency and speed.”
MethodDASNet: Dynamic Activation Sparsity for Neural Network Efficiency Improvement
Yang, Q. X., Mao, J., Wang, Z., & Li, H. H. (2019) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2019.00197 · Cited by 22 · cited 2×
“Through activation sparsification, DasNet [231] reports MAC elimination of 27% for AlexNet and 12% for MobileNet with less than 1% top accuracy degradation.”
MethodAdaptive Tiling: Applying Fixed-size Systolic Arrays To Sparse Convolutional Neural Networks
Kung, H. T., McDanel, B., & Zhang, S. Q. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545462 · Cited by 21 · cited 2×
“It is capable of processing 34.4 frames/s with an energy efficiency of 1.32 TOPS/W. SNAP [309] is another ASIC designed in a 16-nm process that adopted unstructured sparsity for optimization.”
MethodInfluentialExploring the capabilities of mobile devices in supporting deep learning
Chen, Y., Biookaghazadeh, S., & Zhao, M. X. (2019) · ACM/IEEE Symposium on Edge Computing (SEC) · doi:10.1145/3318216.3363316 · Cited by 21
Pruning Filters and Classes: Towards On-Device Customization of Convolutional Neural Networks
Guo, J., & Potkonjak, M. (2017) · doi:10.1145/3089801.3089806 · Cited by 20
CoCoPIE: Making Mobile AI Sweet As PIE --Compression-Compilation Co-Design Goes a Long WayPreprint
Liu, S., Ren, B., Shen, X., & Wang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.06700 · Cited by 20 · cited 1×
“An experimental demonstration of CoCoPIE resulted in improved energy efficiency and inference latency for ResNet50 and VGG-16 models in general-purpose smartphones compared to neural accelerator ASICs and FPGAs [274].”
MethodEdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision ApplicationsPreprint
Maaz, M., Shaker, A., Cholakkal, H., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.10589 · Cited by 20
Lite-CNN: A High-Performance Architecture to Execute CNNs in Low Density FPGAs
Véstias, M. P., Duarte, R. P., de Sousa, J. T., & Neto, H. C. (2018) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2018.00075 · Cited by 19 · cited 1×
“Due to the increased demand for EI, small and high-density FPGAs are now being considered [11], [315], [316], [317].”
BackgroundTF-Replicator: Distributed Machine Learning for Researchers
Buchlovsky, P., Budden, D., Grewe, D., Jones, C., Aslanides, J., Besse, F., Brock, A., Clark, A., Colmenarejo, S. G., Pope, A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00465 · Cited by 19 · cited 1×
“TF replicator is another framework for leveraging distributed ML across TPUs [301].”
Cloud Energy Micro-Moment Data Classification: A Platform Study
Alsalemi, A., Al‐Kababji, A., Himeur, Y., Bensaali, F., & Amira, A. (2020) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1109/ucc48980.2020.00066 · Cited by 19
CoNNa–Hardware accelerator for compressed convolutional neural networks
Struharik, R. J. R., Vukobratović, B. Z., Erdeljan, A. M., & Rakanović, D. M. (2020) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2020.102991 · Cited by 18 · cited 1×
“Skipping zeros from weights and activations may result in reduced performance efficiency [109].”
BackgroundEdge ai: leveraging the full potential of deep learning
Shuvo, M. M. H. (2022) · Studies in Computational Intelligence · doi:10.1007/978-3-031-14748-7_2 · Cited by 18
Self-citationMaximizing Communication Efficiency for Large-scale Training via 0/1 AdamPreprint
Lu, Y., Li, C., Zhang, M., De, C., & He, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06009 · Cited by 18 · cited 1×
“In addition, 1-bit Adam [292], 0/1 Adam [293], and 1-bit LAMB [294] optimizers reduce the communication resource demand in DeepSpeed.”
BackgroundMachine Learning Embedded Smartphone Application for Early-Stage Diabetes Risk Assessment
Shuvo, M. M. H., Ahmed, N., Islam, H., Alaboud, K., Cheng, J., Mosa, A. S. M., & ISLAM, S. K. (2022) · IEEE International Symposium on Medical Measurements and Applications (MeMeA) · doi:10.1109/memea54994.2022.9856420 · Cited by 17 · cited 1×
“Therefore, efficient ML inferences on edge devices are often crucial [27].”
BackgroundSelf-citationCPU/GPU Collaboration Techniques for Transfer Learning on Mobile Devices
Valery, O., Liu, P., & Wu, J. (2017) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads.2017.00069 · Cited by 16 · cited 2×
“For example, features learned in a trained model are transferred to edge nodes that are updated by retraining with local data [354].”
BackgroundJelčicová, Z., & Verhelst, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.03479 · Cited by 16 · cited 1×
“A dynamic pruning technique is reported in [102] to minimize the inference latency of transformer networks while deploying on edge devices.”
MethodHigh performance deep neural network on low cost mobile GPU
Tsung, P.-K., Tsai, S.-F., Pai, A., Lai, S.-J., & Lu, C.-P. (2016) · International Conference on Consumer Electronics · doi:10.1109/icce.2016.7430525 · Cited by 15
Log-quantized stochastic computing for memory and computation efficient DNNs
Sim, H., & Lee, J. (2019) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3287624.3287714 · Cited by 14 · cited 1×
“There are significant efforts to reduce the computation time and latency of SC-based multiplication in DL using logarithmic quantization [179] and differential MAC [180].”
Shang, F., Lai, J., Chen, J., Xia, W., & Liu, H. (2021) · International Conference on Cloud Computing and Big Data Analytics (ICCCBDA) · doi:10.1109/icccbda51879.2021.9442600 · Cited by 14 · cited 1×
“Anomaly detection of electrical equipment based on compressed AlexNet through pruning and quantization can detect the status of the instrument using images [408].”
MethodAn Optimized Hardware Implementation of Deep Learning Inference for Diabetes Prediction
Shuvo, M. M. H., Hassan, O., Parvin, D., Chen, M., & ISLAM, S. K. (2021) · IEEE International Instrumentation and Measurement Technology Conference · doi:10.1109/i2mtc50364.2021.9459794 · Cited by 14 · cited 1×
“Quantization of weights as the nearest power of two can replace the multiplication with faster arithmetic shift operation that significantly reduces the computation [114], [151].”
BackgroundSelf-citationIntel Distribution of OpenVINO Toolkit: A Case Study of Semantic Segmentation
Кустикова, В. Д., Vasiliev, E., Khvatov, A., Kumbrasiev, P., Вихрев, И., Уткин, К., Дудченко, А., & Gladilov, G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-37334-4_2 · Cited by 13
A CGRA based Neural Network Inference Engine for Deep Reinforcement Learning
Liang, M., Chen, M., Wang, Z., & Sun, J. (2018) · IEEE Asia Pacific Conference on Circuits and Systems (APCCAS) · doi:10.1109/apccas.2018.8605639 · Cited by 12 · cited 1×
“For RL, a CGRA architecture is introduced in [328].”
BackgroundGPGPU Accelerated Deep Object Classification on a Heterogeneous Mobile Platform
Rizvi, S. T. H., Cabodi, G., Patti, D., et al. (2016) · Electronics · doi:10.3390/electronics5040088 · Cited by 11 · cited 1×
“Nvidia’s compute unified device architecture (CUDA) is a parallelization framework that can be used for embedded and mobile devices [277].”
MethodADaPT: optimizing CNN inference on IoT and mobile devices using approximately separable 1-D kernels
Maji, P., Bates, D., Chadwick, A., & Mullins, R. (2017) · doi:10.1145/3109761.3109804 · Cited by 11
Low Precision Deep Learning Training on Mobile Heterogeneous Platform
Valery, O., Liu, P., & Wu, J. (2018) · Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP) · doi:10.1109/pdp2018.2018.00023 · Cited by 11 · cited 2×
“Besides, model compression to 8-bit fixed precision quantization [355] results in faster computation.”
BackgroundMethodDesign and Implementation of the Vehicular Camera System using Deep Neural Network Compression
Kim, B., Jeon, Y., Park, H., Han, D., & Baek, Y. (2017) · doi:10.1145/3089801.3089803 · Cited by 10 · cited 1×
“DL-based vehicle detection to assist autonomous driving has been presented in [399], which uses the faster-RCNN network.”
MethodGuihot, H. (2012) · Apress eBooks · doi:10.1007/978-1-4302-4000-6_9 · Cited by 10
Fast and low-power behavior analysis on vehicles using smartphones
Xu, X., Yin, S., & Ouyang, P. (2017) · International Symposium on Next-Generation Electronics · doi:10.1109/isne.2017.7968748 · Cited by 9 · cited 1×
“An RNN model is used in [400] for driving pattern analysis from raw sensor data.”
MethodTowards Deep Learning on Resource-Constrained Robots: A Crowdsourcing Approach with Model Partition
Huai, Z., Ding, B., Wang, H., Geng, M., & Zhang, L. (2019) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/smartworld-uic-atc-scalcom-iop-sci.2019.00194 · Cited by 9 · cited 1×
“DL inference in mobile robots utilizing the optimal model partitioning to incorporate idle robots in proximity shows a 6 × speedup than remote execution in the cloud [373].”
MethodFactorization-Aware Training of Transformers for Natural Language Understanding on the Edge
Saghir, H., Choudhary, S., Eghbali, S., & Chung, C. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1816 · Cited by 9 · cited 1×
“A factorization-aware training is presented in [369] for natural language understanding (NLU) in resource-constraint devices, such as voice assistants.”
BackgroundFaster R-CNN with structured sparsity learning and Ristretto for mobile environment
Nasution, M. A., Chahyati, D., & Fanany, M. I. (2017) · International Conference on Advanced Computer Science and Information Systems · doi:10.1109/icacsis.2017.8355051 · Cited by 8 · cited 1×
“Quantization of only weights to 8-bit integer and output to 16-bit integer can result in 4.16 × storage reduction [121].”
BackgroundInfluentialTable-Based Neural Units: Fully Quantizing Networks for Multiply-Free InferencePreprint
Covell, M., Marwood, D., Baluja, S., & Johnston, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04798 · Cited by 8 · cited 1×
“For example, replacing the multiplication operation with an LUT in [148] or using a hash function to create and train values from the LUT [149] has already been proposed.”
MethodNo Multiplication? No Floating Point? No Problem! Training Networks for Efficient InferencePreprint
Baluja, S., Marwood, D., Covell, M., & Johnston, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09244 · Cited by 7 · cited 1×
“The technique presented in [165] eliminates FLOPs, multiplications, and nonlinear operations during inference.”
MethodBinary Convolutional Neural Network with High Accuracy and Compression Rate
Liu, S., & Zhu, H. (2019) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1145/3377713.3377721 · Cited by 6 · cited 1×
“The BNN model accuracy can be improved in two ways: 1) increasing the number of weights and 2) adopting several binarizations to input activations (IAs) [138].”
MethodLossless Acceleration for Seq2seq Generation with Aggressive Decoding
Ge, T., Xia, H., Sun, X., Chen, S.-Q., & Wei, F. (2022) · arXiv · doi:10.48550/arxiv.2205.10350 · Cited by 6 · cited 1×
“Another efficient and lossless acceleration of seq2seq generation is presented in [382] that exploits aggressive decoding and parallel processing.”
Background
Research works printed but not linked: 30
- Wendelken, 2022, Worldwide and U.S. IoT Cellular Connections Forecast, 2021–2025
- Gupta, ProtoNN: Compressed and accurate KNN for resource-scarce devices, 1331
- Dennis, 2020, EdgeML: Machine learning for resource-constrained edge devices
- Kumar, Resource-efficient machine learning in 2 KB RAM for the Internet of Things, 70, 1935
- 2022, AutoML
- 2022, AutoKeras
- Ryu, BitBlade: Area and energy-efficient precision-scalable neural network accelerator with bitwise summation, 1
- Hojabr, SkippyNN: An embedded stochastic-computing accelerator for convolutional neural networks, 1
- 2022, Jetson TX2 Module
- 2022, Intel® Edison Development Platform
- 2022, Open Neural Network Exchange
- 2022, Deep Learning HDL Toolbox
- 2022, X-CUBE-AI, AI Expansion Pack for STM32CubeMX
- 2022, DNNDK User Guide
- 2022, NVIDIA TensorRT
- 2022, CEVA Deep Neural Network (CDNN)
- 2021, Qualcomm Neural Processing SDK for AI
- 2022, Stratus High-Level Synthesis
- 2022, Deploy Machine Learning Models on Mobile and IoT Devices
- Rasley, DeepSpeed, 3505
- Baines, 2021, FairScale: A general purpose modular PyTorch library for high performance and large scale training
- Dai, BigDL, 50
- 2022, Tensilica DNA Processor Family for on-Device AI
- 2022, DesignWare ARC EV Processors for Embedded Vision
- 2022, AI for the Edge
- 2022, Products, Helping You Bring Local AI to Applications From Prototype to Production
- Shacham, 2022, Pixel Visual Core: Image Processing and Machine Learning on Pixel 2
- Yang, Sparse ReRAM engine: Joint exploration of activation and weight sparsity in compressed neural networks, 236
- Mao, A privacy-preserving deep learning approach for face recognition with edge computing, 1
- Wang, Not just privacy, 2407
Cited by
Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
Wang, X., Tang, Z., Guo, J., Meng, T., Wang, C., Wang, T., & Jia, W. (2025) · ACM Computing Surveys · doi:10.1145/3724420 · Cited by 202
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169
“Moreover, the quadratic complexity of the self-attention mechanism with respect to sequence length exacerbates computational demands, creating severe throughput bottlenecks on edge Central Processing Units (CPUs), Graphics Processing Units (GPUs) or Neural Processing Units (NPUs) [156].”
Review of Lightweight Deep Convolutional Neural Networks
Chen, F., Li, S., Han, J., Ren, F., & Yang, Z. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-10032-z · Cited by 164
“And in [12–19] focuses on the techniques of compacting and accelerating DNN models.”
InfluentialFuture of generative adversarial networks (GAN) for anomaly detection in network security: A review
Lim, W., Yong, K. S. C., Lau, B. T., & Tan, C. C. L. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103733 · Cited by 165
Edge deep learning in computer vision and medical diagnostics: a comprehensive survey
Xu, Y., Khan, T. M., Song, Y., & Meijering, E. H. W. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11033-5 · Cited by 236
Metadata
- Authors
- Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I.
- Affiliations
- University of Missouri (US)
- Texas Tech University (US)
- Venue
- Proceedings of the IEEE
- Published
- 2023-01
- Volume
- 111
- Issue
- 1
- Pages
- 42-91
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W4312910656
- Cited by
- 436
- References
- 491
- Citations by year
- 2026: 159 · 2025: 173 · 2024: 82 · 2023: 22
- FWCI
- 22.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- ThinkTech (Texas Tech University) (submittedVersion, cc-by)
- ThinkTech (Texas Tech University) (submittedVersion)
- Funders
- College of Engineering, University of Missouri, Columbia, MO, USA
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- deep learning inference, edge computing, resource-constrained devices, inference acceleration, algorithm-hardware co-design, neural network accelerators, model compression, Internet of Things, embedded systems, energy-efficient computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.