Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Open accessThe DOI matches its Crossref record; 410 of the 491 references it cites were found in a registry or an index.
Abstract
Successful integration of deep neural networks (DNNs) or deep learning (DL) has resulted in breakthroughs in many areas. However, deploying these highly accurate models for data-driven, learned, automatic, and practical machine learning (ML) solutions to end-user applications remains challenging. DL algorithms are often computationally expensive, power-hungry, and require large memory to process complex and iterative operations of millions of parameters. Hence, training and inference of DL models are typically performed on high-performance computing (HPC) clusters in the cloud. Data transmission to the cloud results in high latency, round-trip delay, security and privacy concerns, and the inability of real-time decisions. Thus, processing on edge devices can significantly reduce cloud transmission cost. Edge devices are end devices closest to the user, such as mobile phones, cyber–physical systems (CPSs), wearables, the Internet of Things (IoT), embedded and autonomous systems, and intelligent sensors. These devices have limited memory, computing resources, and power-handling capability. Therefore, optimization techniques at both the hardware and software levels have been developed to handle the DL deployment efficiently on the edge. Understanding the existing research, challenges, and opportunities is fundamental to leveraging the next generation of edge devices with artificial intelligence (AI) capability. Mainly, four research directions have been pursued for efficient DL inference on edge devices: 1) novel DL architecture and algorithm design; 2) optimization of existing DL methods; 3) development of algorithm–hardware codesign; and 4) efficient accelerator design for DL deployment. This article focuses on surveying each of the four research directions, providing a comprehensive review of the state-of-the-art tools and techniques for efficient edge inference.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These data cannot be managed by relying only on the limited functional services provided by cloud service centers (Ge et al. 2023;Shuvo et al. 2022).”
From Survey: federated learning data security and privacy-preserving in edge-Internet of Things (2024) · cites this paper as Shuvo et al. 2022
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“NLP models, such as Transformer [45] and BERT [224], have around 80% [225] and 93% [226] weight sparsity.”
MethodGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · cited 1×
“Thus, splitting the computation of convolution and dense layer between GPU and CPU can achieve a 60× speedup for object detection using YOLO in the Jetson TK1 board [283].”
MethodDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
(2014) · Cited by 33,419
Research works printed but not linked: 30
- Wendelken, 2022, Worldwide and U.S. IoT Cellular Connections Forecast, 2021–2025
- Gupta, ProtoNN: Compressed and accurate KNN for resource-scarce devices, 1331
- Dennis, 2020, EdgeML: Machine learning for resource-constrained edge devices
- Kumar, Resource-efficient machine learning in 2 KB RAM for the Internet of Things, 70, 1935
- 2022, AutoML
- 2022, AutoKeras
- Ryu, BitBlade: Area and energy-efficient precision-scalable neural network accelerator with bitwise summation, 1
- Hojabr, SkippyNN: An embedded stochastic-computing accelerator for convolutional neural networks, 1
- 2022, Jetson TX2 Module
- 2022, Intel® Edison Development Platform
- 2022, Open Neural Network Exchange
- 2022, Deep Learning HDL Toolbox
- 2022, X-CUBE-AI, AI Expansion Pack for STM32CubeMX
- 2022, DNNDK User Guide
- 2022, NVIDIA TensorRT
- 2022, CEVA Deep Neural Network (CDNN)
- 2021, Qualcomm Neural Processing SDK for AI
- 2022, Stratus High-Level Synthesis
- 2022, Deploy Machine Learning Models on Mobile and IoT Devices
- Rasley, DeepSpeed, 3505
- Baines, 2021, FairScale: A general purpose modular PyTorch library for high performance and large scale training
- Dai, BigDL, 50
- 2022, Tensilica DNA Processor Family for on-Device AI
- 2022, DesignWare ARC EV Processors for Embedded Vision
- 2022, AI for the Edge
- 2022, Products, Helping You Bring Local AI to Applications From Prototype to Production
- Shacham, 2022, Pixel Visual Core: Image Processing and Machine Learning on Pixel 2
- Yang, Sparse ReRAM engine: Joint exploration of activation and weight sparsity in compressed neural networks, 236
- Mao, A privacy-preserving deep learning approach for face recognition with edge computing, 1
- Wang, Not just privacy, 2407
Cited by
Items 401–420 of 420
Wearable Edge AI System for Real-Time Obstructive Sleep Apnea Detection and Severity Classification
Wood, O., Santiesteban, D. A. C., Davis, D. B., Bari, S., & Shuvo, M. M. H. (2026) · IEEE International Midwest Symposium on Circuits and Systems (MWSCAS) · doi:10.1109/mwscas67364.2026.11680800
“Advances in machine learning (ML) and embedded systems now make it possible to deploy such a detection pipeline in real time [7], [8], addressing both the diagnostic access gap and the need for continuous monitoring in high-risk populations.”
Self-citationProfEdge: Efficient Construction of DNN Performance Evaluation Model on Edge Devices
Wang, W., Lu, S., Liu, J., Li, B., Xu, Z., & Wang, S. (2026) · ACM Transactions on Internet Technology · doi:10.1145/3847112
Embedded AI for Subject-Independent Brain Computer Interface using EEG Signals
Meraz, I., Amezaga, M., Islam, M. R., Siddiqua, F., & Shuvo, M. M. H. (2026) · IEEE International Midwest Symposium on Circuits and Systems (MWSCAS) · doi:10.1109/mwscas67364.2026.11680633
“While classification accuracy has been the primary focus of MI-BCI research, comparatively limited attention has been given to the deployment on a resource-constrained hardware platform, which is important for practical applications [17], [18].”
Self-citationHe, Y., Wu, H., Liu, W., et al. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109631
Lightweight Deep Learning Architectures for High-Performance Edge Computing Neural Networks
Singh, S. K., & Kumar, R. (2026) · International Conference on Green Technologies and Sustainable Solutions (GTSS) · doi:10.1109/gtss68743.2026.11679757
SiameseNet+: Classification of HDLSS data with tensor affinities
Cai, H., Wang, Y., Qi, F., Li, J., Wang, Z., Qian, Y., & Cheung, Y. (2026) · Artificial Intelligence · doi:10.1016/j.artint.2026.104623
Biswas, S. (2026) · American Journal of Advanced Technology and Engineering Solutions · doi:10.63125/8pwsg720 · Cited by 1
Energy-Efficient Deep Neural Networks for Sustainable and Intelligent Edge Computing
Fallah, D., Ali, N. J. M., Ahmed, M. A., Shnain, S. K., & Pepa, Y. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39708-9_12
A Review of Lightweight Algorithm‐Driven Intelligent Sensing Technologies for Extreme Environments
Wang, Y., Li, S., Xiao, Z., Tian, L., Yang, C., Song, J., Zheng, L., Zhang, L., & Lawrie, C. H. (2026) · Journal of Sensors · doi:10.1155/js/3919320
AI-Enabled Cybersecurity Monitoring on Industrial IoT Sensor Hardware in Oil and Gas Fields
Chatterjee, S., & Chakrobortty, U. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-30957-0_49
Area–accuracy evaluation of a quantized attention head on a resource-constrained FPGA
Danko, T., & Ádám, N. (2026) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2026.105348
Edge AI for Biomedical Applications: Innovations in Sensing, Computing and Security
Shuvo, M. M. H., Roy, K. C., & Ghose, S. (2026) · Electronics · doi:10.3390/electronics15194373
Self-citationCho, S., & Lee, D. (2026) · Proceedings of the Institution of Mechanical Engineers Part D Journal of Automobile Engineering · doi:10.1177/09544070261490676
Rezwan, S., Khan, U. A., Reisslein, M., Cabrera, J. A., Hofmann, P., & Fitzek, F. H. P. (2026) · IEEE Access · doi:10.1109/access.2026.3734116
“Therefore, extensive research has investigated general principles of AI hardware accelerators [24], [25], [26] as well as edge AI hardware accelerator platforms [27], [28], [29], [30], [31], [32].”
Measurement-Driven Structured Pruning for Energy-Efficient Deep Neural Networks on Edge Devices
Russell, C., & Godinez, J. C. (2026) · IEEE Jornadas Costarricenses de Investigación en Computación e Informática (JoCICI) · doi:10.1109/jocici69279.2026.11701725
“Characterizations of CNNs on edge accelerators such as the EdgeTPU [12] and system-level studies on Raspberry Pi platforms [13] show that inference energy can diverge from FLOP-or parameter-based expectations.”
Real-time YOLO across Cloud, edge, and IoT: architectures, optimisations, and deployment patterns
Attar, H., Ababneh, J., Kalaycıoğlu, A., Solyman, A., Hafez, M., Elhaty, I. A., & Solyman, A. (2026) · Journal of Cloud Computing Advances Systems and Applications · doi:10.1186/s13677-026-00976-5 · Cited by 1
Alunjati, F. A., Supangkat, S. H., Hidayat, F., Musaddad, A., Lullulangi, D. M., & Mahardini, L. (2026) · International Conference on ICT for Smart Society (ICISS) · doi:10.1109/iciss71466.2026.11715495
“While research exists independently on HLS protocols for media streaming and deep learning for object tracking, this paper fills a distinct research gap by integrating network-level HLS streaming optimizations with model-level and data-level enhancements for localized traffic analytics [9,10].”
Karaltı, İ., Ekinci, F., Kumru, E., Sevindik, М., Altındal, Ö. B., Güzel, M. S., & Akata, İ. (2026) · Biology Bulletin · doi:10.1134/s106235902660251x
“In comparative evaluation, deeper and higher-capacity architectures, such as EdgeNeXT (Shuvo et al., 2022), RepVGG (Wang et al., 2024), and DPN (Santos et al., 2021), achieved superior performance in accuracy, F1-score, and AUC, whereas lightweight models, such as MixNet (Zou et al., 2022) and…”
Accuracy and Performance Evaluation of Low-Cost Platforms for Edge Object Detection
Maihach, B. F., & Oyamada, M. S. (2026) · doi:10.1109/sbcci69725.2026.11706745
“Real-time computer vision applications are increasingly moving toward embedded and edge computing platforms, reducing latency, bandwidth usage, network congestion, and privacy risks associated with cloud-centric Artificial Intelligence (AI) applications [1].”
TaCo: Tasks Co-Programming for Accelerating Inference in Scaling Edge Computing
Kong, L., Yang, T., Li, N., Ota, K., & Dong, M. (2025) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2025.3612401 · Cited by 1
“In recent years, there has been an increasing amount of literature on inference accelerating [15] due to the resource limitation of ECN.”
Metadata
- Authors
- Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I.
- Affiliations
- University of Missouri (US)
- Texas Tech University (US)
- Venue
- Proceedings of the IEEE
- Published
- 2023-01
- Volume
- 111
- Issue
- 1
- Pages
- 42-91
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W4312910656
- Cited by
- 436
- References
- 491
- Citations by year
- 2026: 159 · 2025: 173 · 2024: 82 · 2023: 22
- FWCI
- 22.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the IEEE (publishedVersion, cc-by)
- ThinkTech (Texas Tech University) (submittedVersion, cc-by)
- ThinkTech (Texas Tech University) (submittedVersion)
- Funders
- College of Engineering, University of Missouri, Columbia, MO, USA
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- deep learning inference, edge computing, resource-constrained devices, inference acceleration, algorithm-hardware co-design, neural network accelerators, model compression, Internet of Things, embedded systems, energy-efficient computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.