Bringing ai to edge: from deep learning’s perspective

“For instance, training ResNet50 [7] with ImageNet on 8 Nvidia Tesla P100 takes 29 hours [8], where ImageNet dataset [2] has 1.28 M images of 1000 classes for training.”

“Deep Compression uses three steps, prunning , quantization , and Huffman Encoding , to compress over-parameterized DNN models, such as VGG [77], AlexNet [1].”

“For instance, training ResNet50 [7] with ImageNet on 8 Nvidia Tesla P100 takes 29 hours [8], where ImageNet dataset [2] has 1.28 M images of 1000 classes for training.”

“To boost the efficiency of object detection on resource-limited devices, some works strive to reduce the computational complexity of backbone part of object detection models, like Tiny-dsod [58] for DSOD [69], Tiny-SSD [59] for SSD [65], and Tiny-YOLO [57] for YOLO.”

“These models, like BERT [217], are known to be highly complex, even more complicated than DNN models for image classi-fication, and a recent study in [9] raises a concern regarding the environment effect of training complex NLP models.”

“EXPERIMENTAL RESULTS ARE UPON TWO WIDELY-USED BENCHMARK, PASCAL VOC2007 [54] AND MS COCO [55].”

“To improve the performance and training speed, modern DNN models also have other operation layers, like batch normalization layer [38], squeeze-and-excitation layer [39], etc.”

“RCNN [66], Fast RCNN [67], and 3 https://https://www.automl.org/ Faster RCNN [68] are representative examples of two-stage methods.”

“different design patterns in terms of the resource constraint. consideration may not fully utilize the underlying hardware. Fig 13 shows the latency of four state-of-the-art DNN models, Inception v3 [206], ResNet50 [7], MnasNet [53] and MobileNet V3 [45] on three edge devices, Intel Neural Computing Stick 2 [207], Nvi…”

“UMBER OF CHANNELS. HERE WE ONLY REPORT THE RESULTS OF THE SCALING FACTOR=1. Model Year Parameters FLOPs/MACs Accuracy SqueezeNet [41] 2016 1.24M 60.4% Mobilenet [43] 2017 4.2M /569M 70.6% MobilenetV2 [44] 2018 3.4M /300M 72% MobilenetV2-large [44] 2018 6.9M /585M 74.7% MobilenetV3-small [45] 2019 2.5M /56M 67.4% Mobil…”

“Transferability [150]: Knowledge distillation demonstrates better transferability for the small models, i.e., for a given dataset, learning the small model from knowledge distillation outperforms learning it from scratch.”

“To boost the efficiency of object detection on resource-limited devices, some works strive to reduce the computational complexity of backbone part of object detection models, like Tiny-dsod [58] for DSOD [69], Tiny-SSD [59] for SSD [65], and Tiny-YOLO [57] for YOLO.”

“her prominent DNN models targeting resource-limited devices. MobileNet [43] replaces the conventional convolutional operation with more efficient depth-wise separable convolution operation proposed in [46] to reduce the computational cost. Depth-wise separable convolution operation factorizes a conventional k k n convo…”

“MobileNet [43] replaces the conventional convolutional operation with more efficient depth-wise separable convolution operation proposed”

“To boost the efficiency of object detection on resource-limited devices, some works strive to reduce the computational complexity of backbone part of object detection models, like Tiny-dsod [58] for DSOD [69], Tiny-SSD [59] for SSD [65], and Tiny-YOLO [57] for YOLO.”

“AlphaGo [6] is another prominent example which defeated the top professional player in the ancient Chinese game GO, deemed impossible before due to its extremely high complexity.”

“After [140], many efforts are made towards improving the performance of knowledge distillation.”

“2020), researchers around the world have designed many compelling DNN models, applicable to various domains, like nature language processing [199], AI-assisted medicine [200], robots, etc.”

“ShuffleNet [48] deploys group convolution and channel”

“1% GPU/Edge CPU EfficientDet [63] 2020 Two-stage 8.”

“The latest MobileNetV3 [45] combines NAS and NetAdapt [47] to design a more accurate and efficient network architecture.”

“To address these concerns, edge computing is proposed [12].”

“The one-stage approaches can trade off accuracy for efficiency, but the practical deployment on edge devices is still behind the expected performance [70].”

“Another common method to achieve adaptivity is multi-branch which is conceptually analogous to mixture of experts [190] [191], where each branch has some convolution kernels and the multi-branch network selects some branches/convolution kernels per input to conduct inference.”

“GhostNet [51] proposes a ghost module based on an observation that some features in convolutional layers are highly correlated.”

“Different methodologies are devised to approach this goal, like novel light-weight DNN model design, model compression, and the emergent neural architecture search (NAS) [22], all of which facilitate the development and application of DNN models on edge systems.”

“GhostNet [51] proposes a ghost module based on an observation that some features in convolutional layers are highly correlated.”

“Some prior DNN reviews cover one of the three categories, like [25] [26] for model compression, [27] for early works on efficient DNN processing and [28] for NAS1 but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge…”

“Like [218], an efficient point-voxel CNN is proposed for efficient 3D learning, and this model can help to implement 3D AR/VR applications and SLAM [219] of autonomous driving on edge systems.”

“[118] quantize models into 8bit integer and propose a quantize-aware training method to eliminate the accuracy loss caused by quantization.”

“These models, like BERT [217], are known to be highly complex, even more complicated than DNN models for image classi-fication, and a recent study in [9] raises a concern regarding the environment effect of training complex NLP models.”

“Verelst and Tuytelaars [184] use Gumble-softmax [185] to train a pixel-wise gate to identify the region of interest while Chen et al. [186] propose to combine attention with multi-branch technique to improve the capacity of lightweight models.”

“An increasing attention is paid to hardware-aware NAS and some works exploit NAS to design hardware-efficient DNNs. Tan et al. [53] propose a hardware-aware NAS framework, dubbed MnasNet, in which both latency and accuracy are formulated into the reward function of reinforcement learning algorithm…”

“For other related topics, we refer interesting readers to other excellent literature [23], [30], [31].”

“For instance, DNNs are seen to be a pivotal technique for self-driving cars, in which most of executions are subject to highly rigorous real-time constraints [5], [11], but the high and uncertain communication overhead makes it difficult to satisfy the temporal requirement.”

“Moreover, knowledge distillation can also be used to design robust networks [151].”

“Remero et al. [139] adopt knowledge distillation to train a compact model, namely FitNets.”

“In addition, for edge systems, the trade-off between speed and accuracy should be application or context dependent, so a good benchmark and design guide should be developed for practitioners [71] .”

“…Inference-based Layer Rule-based 2016 Hu et al. [88] Inference-based Global Rule-based 2016 SSL [89] Training-based Layer Rule-based 2016 He et al. [90] Inference-based Layer Rule-based 2017 ThiNet [91] Inference-based Layer Rule-based 2017 DeepIoT [92] Training-based Layer Learing-based 2017 DCP…”

“Considering the anticipated emergence of numerous edge intelligence systems in the next years [13], this motivates us to comprehensively review the”

“For instance, training ResNet50 [7] with ImageNet on 8 Nvidia Tesla P100 takes 29 hours [8], where ImageNet dataset [2] has 1.28 M images of 1000 classes for training.”

“UMBER OF CHANNELS. HERE WE ONLY REPORT THE RESULTS OF THE SCALING FACTOR=1. Model Year Parameters FLOPs/MACs Accuracy SqueezeNet [41] 2016 1.24M 60.4% Mobilenet [43] 2017 4.2M /569M 70.6% MobilenetV2 [44] 2018 3.4M /300M 72% MobilenetV2-large [44] 2018 6.9M /585M 74.7% MobilenetV3-small [45] 2019 2.5M /56M 67.4% Mobil…”

“Recently, few works study to employ incremental learning [227] and federated learning [230] [231] with edge systems, but the research in this context is still in its infancy.”

“The successful application of NAS on image classification inspires researchers to explore the potential application of NAS on other CV tasks, like NAS-FPN [157] and NAS-FCOS [159] for object detection and AdversarialNAS [161] for GAN.”

“We have seen some recent efforts towards defining the effective design space for DNN models [209], [210].”

“The off-the-shelf hardware and software cannot efficiently execute those compressed formats, so specialized hardware and software libraries are required to execute sparsified DNN models [83], [84].”

“Some initial works on this topic are called conditional computation [172], [173], where the main objective of conditional computation is to enhance the capacity of a network while not significantly increasing the computational cost.”

“[90] Inference-based Layer Rule-based 2017 ThiNet [91] Inference-based Layer Rule-based 2017 DeepIoT [92] Training-based Layer Learing-based 2017 DCP et al.”

“…hardware architectures demonstrate different degrees of parallelism, the state-of-the-art methods may unnecessarily prune models without performance improvement (e.g., latency reduction) while reducing the capacity of DNN models which is proven to have a significant impact on model’s accuracy [108].”

“In the first DNN decade (2011-2020), researchers around the world have designed many compelling DNN models, applicable to various domains, like nature language processing [199], AI-assisted medicine [200], robots, etc.”

“Moreover, DNN models are found to be vulnerable to adversary attack [202], so security is emerging as another new design concern for ML systems.”

“Other binarized methods are proposed to improve the degraded accuracy caused by binarization like [113], [116], [119].”

“As estimated in [152], MnasNet needs 40,000 GPU hours to search for the optimal network.”

“As identified in literature [149], knowledge distillation provides several benefits for small network models.”

“Verelst and Tuytelaars [184] use Gumble-softmax [185] to train a pixel-wise gate to identify the region of interest while Chen et al. [186] propose to combine attention with multi-branch technique to improve the capacity of lightweight models.”

“Chen and Ran in [24] discuss the diverse applications when deep learning techniques are integrated into edge computing.”

“For other related topics, we refer interesting readers to other excellent literature [23], [30], [31].”

“…Inference-based Layer Rule-based 2018 SFP [94] Training-based Layer Rule-based 2018 Huang et al. [95] Training-based Global Rule-based 2018 AMC [96] Inference-based Layer Learning-based 2018 NetAdapt [47] Inference-based Layer Learning-based 2018 Gate Decorator [97] Inference-based Global…”

“Some early work has shown that using FP16 to train DNN models can reduce the computational cost while retaining accuracy [109].”

“BranchyNet [189] uses early-exit to skip some layers for easy input images, where a threshold is given to evaluate the confidence of predictions from early-exit.”

“2 † Latency Image Classification Yes FBNet [155] LUT gradient Latency Image Classification Yes DenseNAS [156] LUT gradient 3 Latency Image Classification Yes NAS-FPN [157] RL Object Detection Yes* MnasFPN [158] Measure RL Latency Object Detection Yes NAS-FCOS [159] RL Object Detection No Auto-FPN [160] gradient Object…”

“Wu et al. [110] propose a unified quantization framework, which improves the quantization performance by minimizing the estimation error of each layer’s response.”

“Some recent works study to compress the complex NLP models such that they can be deployed on resource-constrained edge systems [33] [34] [35].”

“[93] Inference-based Layer Rule-based 2018 SFP [94] Training-based Layer Rule-based 2018 Huang et al.”

“Recently, few works study to employ incremental learning [227] and federated learning [230] [231] with edge systems, but the research in this context is still in its infancy.”

“Some prior DNN reviews cover one of the three categories, like [25] [26] for model compression, [27] for early works on efficient DNN processing and [28] for NAS 1 but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient…”

“…behind [118] 8-bit Yes Yes 2018 Bi-Real-Net [119] 1-bit Yes Yes 2018 LQ-Net [120] Arbitrary Yes Yes 2018 TQT [121] 8-bit Yes Yes 2019 HAQ [122] (1 ∼ 8)-bit § Yes Yes 2019 HAWQ [123] 2/4-bit † Yes Yes 2019 Jung et al. [124] (2 ∼ 4)-bit Yes Yes 2019 Zhuang et al. [125] 2-bit Yes Yes 2020…”

“Instead of quantizing all weights in a DNN, IQN [115] adopts a group-wise quantization method to gradually quantize weights of a DNN model.”

“…Layer Learning-based 2018 Gate Decorator [97] Inference-based Global Rule-based 2019 LFPC [98] Inference-based Layer Learning-based 2020 HRank [99] Inference-based Layer Rule-based 2020 • Inference-based : The prune method reduces the redundant channels from a pre-trained model according to…”

“On top of TWNs, Zhu et al. [117] further introduce two independent quantization scaling factors for positive and negative weights in each layer, respectively, to improve accuracy of ternary quantization.”

“Some prior DNN reviews cover one of the three categories, like [25] [26] for model compression, [27] for early works on efficient DNN processing and [28] for NAS 1 but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient…”

“[82] heuristic Energy 2017 Zhang et al.”

“Inspired by [114], Li et al. [112] propose ternary weight networks (TWNs) with weights constrained to {− 1 , 0 , +1 } , which minimizes the Euclidian distance between full precision weights W and the ternary weights W t along with a scaling factor α .”

“Anwar et al. [101] prune a DNN model at feature level, kernel level, and intra-kernel level (i.e., weight), where they deploy evolutionary algorithm to find the best combination of different pruning granularities.”

“The academia and industry have a consensus that the breakthroughs in hardware architecture will bring DNNs to a new level and boost DNN adoption [15] [205].”

“To boost the efficiency of object detection on resource-limited devices, some works strive to reduce the computational complexity of backbone part of object detection models, like Tiny-dsod [58] for DSOD [69], Tiny-SSD [59] for SSD [65], and Tiny-YOLO [57] for YOLO.”

“The max pooling [37] and the average pooling are the two common pooling methods.”

“Combined with quantization, Polino et al. [143] introduce the quantized distillation , which leverages distillation during the training process by incorporating knowledge distillation loss.”

“SkipNet [196] uses previous layer’s activation to determine whether the subsequent layer is required for the inference, where RNN and RL are used to control the executed blocks .”

“[56] implements an object detection model by using SequeezeNet [41] as the backbone to improve the efficiency while not significantly compromising accuracy.”

“Another common method to achieve adaptivity is multi-branch which is conceptually analogous to mixture of experts [190] [191], where each branch has some convolution kernels and the multi-branch network selects some branches/convolution kernels per input to conduct inference.”

“…Layer Learing-based 2017 DCP et al. [93] Inference-based Layer Rule-based 2018 SFP [94] Training-based Layer Rule-based 2018 Huang et al. [95] Training-based Global Rule-based 2018 AMC [96] Inference-based Layer Learning-based 2018 NetAdapt [47] Inference-based Layer Learning-based 2018…”

“[177] Channels Gradient 2019 SlimmableNet [178] Channels Gradient 2019 FBS [176] Channels Gradient 2018 CGNets [187] Channels Gradient 2019 AdaptiveNN [188] Early-exit Gradient 2017 BranchyNet [189] Early-exit Gradient 2017 D2NN [194] Multi-branch RL 2018 Condconv [193] Multi-branch Gradient 2019 HydraNets [192] Multi…”

“Pruning is the well-studied topic in model compression and many methods have been proposed to advance the network pruning [103].”

“HAWQ [123] also considers to quantize a full-precision model into a mix-precision model where the quantized precision is determined for each layer based on Hessian matrix.”

“Zhang et al. [80] formulate the weight-pruning problem as a non-convex problem which can be solved by alternating direction method of multipliers (ADMM) method [81].”

“…RL 2018 Condconv [193] Multi-branch Gradient 2019 HydraNets [192] Multi-branch Gradient 2018 SkipNet [196] Layers RNN/RL 2018 BlockDrop [197] Layers RL 2018 SBnet [181] Attention Gradient 2018 Verelst and Tuytelaars [184] Attention Gradient 2020 Chen et al. [186] Attention Gradient…”

“Scope of this paper : Edge computing is an emergent research topic, consisting of many interesting and challenging research problems, like edge caching [30] and computation offloading [31], [32].”

“…of service (QoS) of applications without rigorously temporal requirement, like voice recognition, face recognition, machine translation, etc [20] and on the other hand may lead to catastrophic consequence for those with rigorously temporal requirement, (i.e., real-time requirement), like…”

“as follows: IWˇI (B) = IB (4) where Iis the input image and B2f 1;+1gis the binarized weight. Other binarized methods are proposed to improve the degraded accuracy caused by binarization like [113], [116], [119]. XOR-Net [126] takes the implementation of BNNs into account and simplifies the number of instructions used…”

“Many companies implement their DNN inference on servers to provide convenient services for their customers such as voice-assistant, machine translation, image retrieval, etc [10].”

“Adaptive Scope Training Year RNP [175] Channels RL 2017 Benjordi et al. [177] Channels Gradient 2019 SlimmableNet [178] Channels Gradient 2019 FBS [176] Channels Gradient 2018 CGNets [187] Channels Gradient 2019 AdaptiveNN [188] Early-exit Gradient 2017 BranchyNet [189] Early-exit Gradient 2017 D 2…”

“…LQ-Net [120] Arbitrary Yes Yes 2018 TQT [121] 8-bit Yes Yes 2019 HAQ [122] (1 ∼ 8)-bit § Yes Yes 2019 HAWQ [123] 2/4-bit † Yes Yes 2019 Jung et al. [124] (2 ∼ 4)-bit Yes Yes 2019 Zhuang et al. [125] 2-bit Yes Yes 2020 XOR-Net [126] 2-bit Yes Yes 2020 a / b -bit † denotes the method quantizes the…”

“SlimmableNet [178], [179], also a channel adaptive method, proposes a method to train a network with different channel width configurations and the network can actively vary its channel configuration on the fly.”

“…Inference-based Layer Rule-based 2018 SFP [94] Training-based Layer Rule-based 2018 Huang et al. [95] Training-based Global Rule-based 2018 AMC [96] Inference-based Layer Learning-based 2018 NetAdapt [47] Inference-based Layer Learning-based 2018 Gate Decorator [97] Inference-based Global…”

“In addition, as shown in [170], the same neural architecture demonstrates up to 62x difference on modern mobile devices in terms of inference time.”

“…Inference-based Layer Rule-based 2016 Hu et al. [88] Inference-based Global Rule-based 2016 SSL [89] Training-based Layer Rule-based 2016 He et al. [90] Inference-based Layer Rule-based 2017 ThiNet [91] Inference-based Layer Rule-based 2017 DeepIoT [92] Training-based Layer Learing-based 2017 DCP…”

“SACT [182] exploits adaptive computation time (ACT) technique [183] to vary the computation per spatial location of input images.”

“SACT [182] exploits adaptive computation time (ACT) technique [183] to vary the computation per spatial location of input images.”

“Unfortunately, the majority of DNN models are designed to pursue better [62] 2019 Two-stage 320 × 320 - 4.61M/ 75.1% - GPU/Edge CPU EfficientDet [63] 2020 Two-stage - 8.1M* 11B*/ - 43.0”

“Li et al. [61] replace the heavy-head in RCNN framework to speed up execution.”

“…accelerators are proposed and many companies are dedicated to develop specific hardware architectures to speed up the execution of DNN models, such as application-specific integrated circuits (ASICs) Tensor Processing Unit (TPU) [15] and DianNao family [16], and FPGA-based accelerators ESE [17] [18].”

“Some recent works study to compress the complex NLP models such that they can be deployed on resource-constrained edge systems [33] [34] [35].”

“[165] present ChamNet which deploys evolution algorithm (EA) and three predictors, i.”

“Pruning methods Scope Process Year Li et al. [87] Inference-based Layer Rule-based 2016 Hu et al. [88] Inference-based Global Rule-based 2016 SSL [89] Training-based Layer Rule-based 2016 He et al. [90] Inference-based Layer Rule-based 2017 ThiNet [91] Inference-based Layer Rule-based 2017 DeepIoT [92] Training-based…”

“The successful application of NAS on image classification inspires researchers to explore the potential application of NAS on other CV tasks, like NAS-FPN [157] and NAS-FCOS [159] for object detection and AdversarialNAS [161] for GAN.”

“Feature Boosting and Suppression (FBS) [176] exploits channel saliency to judge which channels can be skipped, where a low overhead predictor is presented to predict channel saliency according to feature maps of the previous layer.”

“On one hand, edge systems can update models locally by using incremental learning [226] [227], where techniques in [228] are deployed to improve the accuracy for new data and infer unknown classes.”

“Different from above-mentioned approaches which conduct non-structure pruning directly on pre-trained networks, NeST [79], that is inspired by the development of human brain, adopts a grow-prune scheme, where NeST first makes a sparse seed DNN model bigger and more complex and then prunes some…”

“As a result, the huge memory requirement prohibits DNN models from implementing on memory-limited edge devices [127], [128].”

“** Auto-FPN incorporate hardware-agnostic resource constraint like FLOPs, which cannot translate the runtime complexity upon target edge systems.”

“[220] recently present a method to compress generative adversarial nets (GAN) [221].”

“Especially, in future edge era, more and more customized DNN models for specific tasks will be developed and implemented on different hardware platforms, e.g., edge systems with intermittent power supply [169].”

“Pruning methods Scope Process Year Li et al. [87] Inference-based Layer Rule-based 2016 Hu et al. [88] Inference-based Global Rule-based 2016 SSL [89] Training-based Layer Rule-based 2016 He et al. [90] Inference-based Layer Rule-based 2017 ThiNet [91] Inference-based Layer Rule-based 2017 DeepIoT [92] Training-based…”

“…[193] Multi-branch Gradient 2019 HydraNets [192] Multi-branch Gradient 2018 SkipNet [196] Layers RNN/RL 2018 BlockDrop [197] Layers RL 2018 SBnet [181] Attention Gradient 2018 Verelst and Tuytelaars [184] Attention Gradient 2020 Chen et al. [186] Attention Gradient 2020 different from [192],…”

“Moreover, to achieve an optimal design for future edge intelligence systems, NAS may need to take advantage of other model compression techniques like network pruning, quantization, and knowledge distillation, to have a holistic and efficient design paradigm, e.g., [138].”

“Li et al. [167] consider both latency and accuracy as their NAS objectives and use the concept of partial order pruning to reduce the search space such that it can significantly reduce the searching time and achieve a good trade-off between accuracy and latency.”

“[177] Channels Gradient 2019 SlimmableNet [178] Channels Gradient 2019 FBS [176] Channels Gradient 2018 CGNets [187] Channels Gradient 2019 AdaptiveNN [188] Early-exit Gradient 2017 BranchyNet [189] Early-exit Gradient 2017 D2NN [194] Multi-branch RL 2018 Condconv [193] Multi-branch Gradient 2019 HydraNets [192] Multi…”

“The first decade of 21 th century witnessed the emergence of system-level design methodologies [208] for multicore system design.”

“…Gradient 2018 SkipNet [196] Layers RNN/RL 2018 BlockDrop [197] Layers RL 2018 SBnet [181] Attention Gradient 2018 Verelst and Tuytelaars [184] Attention Gradient 2020 Chen et al. [186] Attention Gradient 2020 different from [192], [193], formalize a network as a directly acyclic graph…”

“We need metrics like Roofline [211] to guide the direction in finding efficient network upon a target platform and a standard benchmark which can fairly and quantitatively evaluate various DNN models on new hardware accelerators, like MLPerf [212] and ParaDNN [213].”

“As shown in [29], around 50 papers were uploaded to Arxiv per day in 2018, discussing DNN related topics, i.e., in total more than 15000 papers per year.”

“…Channels RL 2017 Benjordi et al. [177] Channels Gradient 2019 SlimmableNet [178] Channels Gradient 2019 FBS [176] Channels Gradient 2018 CGNets [187] Channels Gradient 2019 AdaptiveNN [188] Early-exit Gradient 2017 BranchyNet [189] Early-exit Gradient 2017 D 2 NN [194] Multi-branch RL 2018…”

“[135] study the effect of quantization on the DNN robustness and propose the Defensive Quantization (DQ) that addresses the robust issue of quantized models, where DQ can maintain adversarial”

“…Layer Rule-based 2017 ThiNet [91] Inference-based Layer Rule-based 2017 DeepIoT [92] Training-based Layer Learing-based 2017 DCP et al. [93] Inference-based Layer Rule-based 2018 SFP [94] Training-based Layer Rule-based 2018 Huang et al. [95] Training-based Global Rule-based 2018 AMC…”

“2 † Latency Image Classification Yes FBNet [155] LUT gradient Latency Image Classification Yes DenseNAS [156] LUT gradient 3 Latency Image Classification Yes NAS-FPN [157] RL Object Detection Yes* MnasFPN [158] Measure RL Latency Object Detection Yes NAS-FCOS [159] RL Object Detection No Auto-FPN [160] gradient Object…”

“In addition, some edge systems are energy-constrained, battery powered or supplied by sustainable energy, like solar energy (intermittent computing [171]), and in these cases energy variance will change the system’s status, such as scaling down frequency and power which affects the inference time…”

“SpArSe [162] targets networks which can be fit into micro-controllers which have small memory footprint and less computation capability, where NAS and pruning techniques are combined to design small-memory networks.”

“We have seen some recent efforts towards defining the effective design space for DNN models [209], [210].”

“Some research strives to have a hardware-aware NAS, like FNAS [204], which directly incorporates resource constraints during implementing DNNs on FPGAs.”

“…2017 Polino et al. [143] From logits Single 2018 Ravi et al. [144] From logits Single 2019 Li et al. [145] From logits Single 2020 Chung et al. [146] From intermediates Single 2020 Liu et al. [147] From intermediates Multiple 2020 † Following [140], we use logits to denote the knowledge from…”

“The successful application of NAS on image classification inspires researchers to explore the potential application of NAS on other CV tasks, like NAS-FPN [157] and NAS-FCOS [159] for object detection and AdversarialNAS [161] for GAN.”

“To boost the efficiency of object detection on resource-limited devices, some works strive to reduce the computational complexity of backbone part of object detection models, like Tiny-dsod [58] for DSOD [69], Tiny-SSD [59] for SSD [65], and Tiny-YOLO [57] for YOLO.”

“Zhang et al. [125] adopt an auxiliary full-precision model to facilitate the training of its quantized counterparts.”

“NetAdapt [47] proposes an automated framework to prune filters in different layers such that the pruned model can be adapted to a target platform.”

“[136] consider to quantize network with the objective of reducing energy consumption.”

“MnasFPN [158] is an extension of MnasNet which, instead of searching classification model, targets directly searching a network for object detection.”

“Gschwend et al. [42] develop a variant of SqueezeNet and implement it on an FPGA.”

“Some initial works on this topic are called conditional computation [172], [173], where the main objective of conditional computation is to enhance the capacity of a network while not significantly increasing the computational cost.”

“However, due to high communication overhead, on-time model update and possible leakage of confidential data, some edge systems prefer to train the model locally, i.e., on-device learning [222].”

“Therefore, some studies [120], [124], [125] aim to effectively train a low-precision, compact model.”

“orm SqueezeDet [56] 2017 One-stage 1242 375 - 9.7B/ - - GPU Tiny-YOLO [57] 2018 One-stage 416 416 15.12M 3.49B/ 57.1% - GPU Tiny-DSOD [58] 2018 One-stage 300 300 1.15M 1.12B/ 72.1% 23.2% GPU Tiny-SSD [59] 2018 One-stage 300 300 1.13M /571.09M 61.3% - GPU Pelee [60] 2018 One-stage 320 320 5.98M 1.21B/ 70.9% 22.4% GPU/E…”

“Yao et al. [92] train an recurrent neural network (RNN) to determine the dropout probability of each filters and prune the network layer by layer according to drop probability.”

“RCNAS [168] formulates the resource-constrained neural architecture search problem as a submodularity function problem which is known to be NP-Hard but has good heuristic algorithms to approximately solve this problem.”

“Other binarized methods are proposed to improve the degraded accuracy caused by binarization like [113], [116], [119].”

“Moreover, to achieve an optimal design for future edge intelligence systems, NAS may need to take advantage of other model compression techniques like network pruning, quantization, and knowledge distillation, to have a holistic and efficient design paradigm, e.g., [138].”