CoChat

Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review

Open access

The DOI matches its Crossref record; 408 of the 491 references it cites were found in a registry or an index.

Abstract

Successful integration of deep neural networks (DNNs) or deep learning (DL) has resulted in breakthroughs in many areas. However, deploying these highly accurate models for data-driven, learned, automatic, and practical machine learning (ML) solutions to end-user applications remains challenging. DL algorithms are often computationally expensive, power-hungry, and require large memory to process complex and iterative operations of millions of parameters. Hence, training and inference of DL models are typically performed on high-performance computing (HPC) clusters in the cloud. Data transmission to the cloud results in high latency, round-trip delay, security and privacy concerns, and the inability of real-time decisions. Thus, processing on edge devices can significantly reduce cloud transmission cost. Edge devices are end devices closest to the user, such as mobile phones, cyber–physical systems (CPSs), wearables, the Internet of Things (IoT), embedded and autonomous systems, and intelligent sensors. These devices have limited memory, computing resources, and power-handling capability. Therefore, optimization techniques at both the hardware and software levels have been developed to handle the DL deployment efficiently on the edge. Understanding the existing research, challenges, and opportunities is fundamental to leveraging the next generation of edge devices with artificial intelligence (AI) capability. Mainly, four research directions have been pursued for efficient DL inference on edge devices: 1) novel DL architecture and algorithm design; 2) optimization of existing DL methods; 3) development of algorithm–hardware codesign; and 4) efficient accelerator design for DL deployment. This article focuses on surveying each of the four research directions, providing a comprehensive review of the state-of-the-art tools and techniques for efficient edge inference.

Overview

Integrity signals

Registry

Crossref · 10.1109/jproc.2022.3226481

Cites

408 of 408 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 30

  1. Wendelken, 2022, Worldwide and U.S. IoT Cellular Connections Forecast, 2021–2025
  2. Gupta, ProtoNN: Compressed and accurate KNN for resource-scarce devices, 1331
  3. Dennis, 2020, EdgeML: Machine learning for resource-constrained edge devices
  4. Kumar, Resource-efficient machine learning in 2 KB RAM for the Internet of Things, 70, 1935
  5. 2022, AutoML
  6. 2022, AutoKeras
  7. Ryu, BitBlade: Area and energy-efficient precision-scalable neural network accelerator with bitwise summation, 1
  8. Hojabr, SkippyNN: An embedded stochastic-computing accelerator for convolutional neural networks, 1
  9. 2022, Jetson TX2 Module
  10. 2022, Intel® Edison Development Platform
  11. 2022, Open Neural Network Exchange
  12. 2022, Deep Learning HDL Toolbox
  13. 2022, X-CUBE-AI, AI Expansion Pack for STM32CubeMX
  14. 2022, DNNDK User Guide
  15. 2022, NVIDIA TensorRT
  16. 2022, CEVA Deep Neural Network (CDNN)
  17. 2021, Qualcomm Neural Processing SDK for AI
  18. 2022, Stratus High-Level Synthesis
  19. 2022, Deploy Machine Learning Models on Mobile and IoT Devices
  20. Rasley, DeepSpeed, 3505
  21. Baines, 2021, FairScale: A general purpose modular PyTorch library for high performance and large scale training
  22. Dai, BigDL, 50
  23. 2022, Tensilica DNA Processor Family for on-Device AI
  24. 2022, DesignWare ARC EV Processors for Embedded Vision
  25. 2022, AI for the Edge
  26. 2022, Products, Helping You Bring Local AI to Applications From Prototype to Production
  27. Shacham, 2022, Pixel Visual Core: Image Processing and Machine Learning on Pixel 2
  28. Yang, Sparse ReRAM engine: Joint exploration of activation and weight sparsity in compressed neural networks, 236
  29. Mao, A privacy-preserving deep learning approach for face recognition with edge computing, 1
  30. Wang, Not just privacy, 2407

Cited by

See all 436

Metadata

Authors
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I.
Affiliations
  • University of Missouri (US)
  • Texas Tech University (US)
Venue
Proceedings of the IEEE
Published
2023-01
Volume
111
Issue
1
Pages
42-91
ISSN
  • 0018-9219 (ISSN-L)
  • 1558-2256
OpenAlex
W4312910656
Cited by
436
References
491
Citations by year
2026: 159 · 2025: 173 · 2024: 82 · 2023: 22
FWCI
22.67
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the IEEE (publishedVersion, cc-by)
  • ThinkTech (Texas Tech University) (submittedVersion, cc-by)
  • ThinkTech (Texas Tech University) (submittedVersion)
Funders
College of Engineering, University of Missouri, Columbia, MO, USA
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Keywords
deep learning inference, edge computing, resource-constrained devices, inference acceleration, algorithm-hardware co-design, neural network accelerators, model compression, Internet of Things, embedded systems, energy-efficient computing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.