Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges
The DOI matches its Crossref record; 61 of the 67 references it cites were found in a registry or an index.
Abstract
In recent years, deep neural networks (DNNs) have received increased attention, have been applied to different applications, and achieved dramatic accuracy improvements in many tasks. These works rely on deep networks with millions or even billions of parameters, and the availability of graphics processing units (GPUs) with very high computation capability plays a key role in their success. For example, Krizhevsky et al. achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully connected layers. Usually, it takes two to three days to train the whole model on the ImagetNet data set with an NVIDIA K40 machine. In another example, the top face-verification results from the Labeled Faces in the Wild (LFW) data set were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally connected, and fully connected layers. It is also very time-consuming to train such a model to obtain a reasonable performance. In architectures that only rely on fully connected layers, the number of parameters can grow to billions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [26]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Recently, more state-of-the-art architectures are used as baseline models in many works, including network in networks [67], VGGNets [68], and ResNets [69].”
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“[1] achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully-connected layers.”
MethodInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The work in [51] introduced a KD compression framework, which eased the training of deep networks by following a student-teacher paradigm, in which the student was penalized according to a softened version of the teacher’s output.”
BackgroundDeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · cited 1×
“example is the top face verification results on the Labeled Faces in the Wild (LFW) dataset were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally-connected, and fully-connected layers [2], [3].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“works that directly train CNNs with binary weights, for instance, BinaryConnect [13], BinaryNet [14] and XNORNetworks [15].”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
Method
Cited by
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2018) · arXiv · doi:10.48550/arxiv.1809.02165 · Cited by 2,965
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01247-4 · Cited by 2,965
Deep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Federated Learning via Over-the-Air Computation
Yang, K., Jiang, T., Shi, Y., & Ding, Z. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2961673 · Cited by 1,084
“To address this issue, there is a growing body of recent works to reduce the storage overhead, time and power consumption in the inference process using the model compression methods via hardware and software co-design [7], [8].”
MethodModel Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
Deng, B. L., Li, G., Han, S., Shi, L., & Xie, Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2976475 · Cited by 996
“…without compression [14], a certain category of compression approaches [15], or a rapid overview of multiple categories on only the algorithm side [16], [17], we present a more systematic summary on most existing methodologies, show more detailed descriptions and comparisons for the individual or…”
MethodEdge artificial intelligence for 6g: vision, enabling technologies, and applications
Letaief, K. B., Shi, Y., Lu, J., & Lu, J. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126076 · Cited by 829
“1) Edge Device Distributed Inference: Enormous efforts on TinyML with DL model compression and neural network architecture search have been conducted to enable low-latency and energy-efficient model inference on a single device with limited storage and computation resources [220].”
BackgroundZhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774
“For other techniques, we recommend the comprehensive review in [191].”
MethodEfficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
Menghani, G. (2023) · ACM Computing Surveys · doi:10.1145/3578938 · Cited by 605
A systematic review on overfitting control in shallow and deep neural networks
Bejani, M. M., & Ghatee, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09975-1 · Cited by 573
Xu, M., Ng, W. C., Lim, W. Y. B., Kang, J., Xiong, Z., Niyato, D., Yang, Q., Shen, X., & Miao, C. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3221119 · Cited by 564
Communication-efficient edge ai: algorithms and systems
Shi, Y., Yang, K., Jiang, T., Zhang, J., & Letaief, K. B. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3007787 · Cited by 488
“Therefore, model compression approaches [27], [28] have attracted much attention for deploying DNN models at network edges.”
MethodUnleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services
Xu, M., Du, H., Niyato, D., Kang, J., Xiong, Z., Mao, S., Han, Z., Jamalipour, A., Kim, D. I., Shen, X., et al. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3353265 · Cited by 449
“increasingly complex, model compression techniques are becoming more important to reduce service latency and resource consumption in provisioning AIGC services [230].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“Four prominent compression techniques, such as parameter pruning, low-rank approximation, compact CNN design, and knowledge distillation (KD), are summarized in [6].”
MethodModel complexity of deep learning: a survey
Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J. (2021) · Knowledge and Information Systems · doi:10.1007/s10115-021-01605-0 · Cited by 402
Structured Pruning for Deep Convolutional Neural Networks: A Survey
He, Y., & Xiao, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3334614 · Cited by 363
Stronger generalization bounds for deep nets via a compression approachPreprint
Arora, S., Ge, R., Neyshabur, B., & Zhang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05296 · Cited by 655
“The idea of compressing trained deep nets is very popular for low-power applications; for a survey see [Cheng et al., 2018].”
BackgroundEfficient multi-objective neural architecture search via Lamarckian evolution
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1804.09081 · Cited by 311
“Note that these operators could easily be extended by sophisticated methods for compressing neural networks (Han et al., 2016; Cheng et al., 2018).”
MethodYao, S., Guan, R., Huang, X., Li, Z., Sha, X., Yue, Y., Lim, E. G., Seo, H., Man, K. L., Zhu, X., et al. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3307157 · Cited by 265
A comprehensive review of model compression techniques in machine learning
Dantas, P. V., Sabino da Silva, W., Cordeiro, L. C., & Carvalho, C. B. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05747-w · Cited by 261
“At the heart of SqueezeNet is the use of fire modules, which are small, carefully designed CNN that drastically reduce the number of parameters without affecting performance [21, 172].”
InfluentialPruning by explaining: A novel criterion for deep neural network pruning
Yeom, S.-K., Seegerer, P., Lapuschkin, S., Binder, A., Wiedemann, S., Müller, K.-R., & Samek, W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107899 · Cited by 263
Zhang, S., Yang, Y., Chen, C., Zhang, X., Leng, Q., & Zhao, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121692 · Cited by 251
End-Edge-Cloud Collaborative Computing for Deep Learning: A Comprehensive Survey
Wang, Y., Yang, C., Lan, S., Zhu, L., & Zhang, Y. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3393230 · Cited by 237
“While Cheng et al. [248] have given some suggestions to select compression methods for a particular task, differences in hardware platforms used for experiments can complicate the unified evaluation of various compression methods.”
A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification
Rokh, B., Azarpeyvand, A., & Khanteymoori, A. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3623402 · Cited by 232
Methods for Pruning Deep Neural Networks
Vadera, S., & Ameen, S. (2022) · IEEE Access · doi:10.1109/access.2022.3182659 · Cited by 223
“[196], [197], Lebedev and Lempitsky [50], 2019-2021 Elsken et al.”
BackgroundCapra, M., Bussolino, B., Marchisio, A., Masera, G., Martina, M., & Shafique, M. (2020) · IEEE Access · doi:10.1109/access.2020.3039858 · Cited by 222
“During the last few years, therefore, there has been a big push towards the research of methods to compress the models of NNs without affecting the achieved accuracy [198].”
BackgroundAziz, L., Haji Salam, M. S. B., Sheikh, U. U., & Ayub, S. (2020) · IEEE Access · doi:10.1109/access.2020.3021508 · Cited by 204
“Network acceleration [244]–[248] and the design of a compact, lightweight network in the field of object detection is one of the new and growing research areas [235], [249]–[253].”
BackgroundBottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems
Shao, J., & Zhang, J. (2020) · 2020 IEEE International Conference on Communications Workshops (ICC Workshops) · doi:10.1109/iccworkshops49005.2020.9145068 · Cited by 200
“A common method to solve this problem is to transmit the raw data to be processed at the cloud or edge computing platforms [3], [4].”
MethodAhmad, S., Shakeel, I., Mehfuz, S., & Ahmad, J. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100568 · Cited by 197
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., & Winslett, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00413 · Cited by 190
Automatic Modulation Classification: A Deep Architecture Survey
Huynh-The, T., Pham, Q.-V., Nguyen, T.-V., Nguyen, T. T., Ruby, R., Zeng, M., & Kim, D.-S. (2021) · IEEE Access · doi:10.1109/access.2021.3120419 · Cited by 183
“Some compression and acceleration techniques [147] can be applied to speed up the training (or learning) and prediction (or inferencing) stages, which are generally categorized into four groups: network pruning, transfer learning, low-rank factorization, and knowledge distillation [148].”
BackgroundA survey on deep learning for cybersecurity: Progress, challenges, and opportunities
Macas, M., Wu, C., & Fuertes, W. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109032 · Cited by 181
Bringing ai to edge: from deep learning’s perspective
Liu, D., Kong, H., Luo, X., Liu, W., & Subramaniam, R. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.04.141 · Cited by 171
“Some prior DNN reviews cover one of the three categories, like [25] [26] for model compression, [27] for early works on efficient DNN processing and [28] for NAS 1 but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient…”
BackgroundReview of Lightweight Deep Convolutional Neural Networks
Chen, F., Li, S., Han, J., Ren, F., & Yang, Z. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-10032-z · Cited by 164
“And in [12–19] focuses on the techniques of compacting and accelerating DNN models.”
Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
Wang, E., Davis, J. J., Zhao, R., Ng, H.-C., Niu, X., Luk, W., Cheung, P. Y. K., & Constantinides, G. A. (2020) · ACM Computing Surveys · doi:10.1145/3309551 · Cited by 166
Communication-Computation Trade-off in Resource-Constrained Edge Inference
Shao, J., & Zhang, J. (2020) · IEEE Communications Magazine · doi:10.1109/mcom.001.2000373 · Cited by 162
“There have been lots of interests in DNN model compression [4], which effectively reduces the memory footprint and computational cost of DNNs.”
BackgroundMethodInfluentialDeep Learning on Mobile and Embedded Devices: State-of-the-art, Challenges, and Future Directions
Chen, Y., Zheng, B., Zhang, Z., Wang, Q., Shen, C., & Zhang, Q. (2021) · ACM Computing Surveys · doi:10.1145/3398209 · Cited by 159
“Reference [25] briefly introduced early works on compression and acceleration methods for deep neural network models.”
BackgroundTrakadas, P., Simoens, P., Gkonis, P., Sarakis, L., Angelopoulos, A., Ramallo-González, A. P., Skarmeta, A., Trochoutsos, C., Calvο, D., Pariente, T., et al. (2020) · Sensors · doi:10.3390/s20195480 · Cited by 155
“In this domain, deep learning (DL) is the dominant branch today, yet DL comes with its own problems: it requires domain knowledge to select the appropriate machine learning pipeline and it is very resource hungry in terms of computing and storage resources [36,43,44].”
BackgroundKraft, M., Piechocki, M., Ptak, B., & Walas, K. (2021) · Remote Sensing · doi:10.3390/rs13050965 · Cited by 187
“Model compression by quantisation, model pruning and compression is an area of active research, with numerous novel ideas proposed [36].”
Sparse low rank factorization for deep neural network compression
Swaminathan, S., Garg, D., Kannan, R., & Andres, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.035 · Cited by 147
Cattle detection and counting in UAV images based on convolutional neural networks
Shao, W. Q., Kawakami, R., Yoshihashi, R., You, S., Hidemichi, K., & Naemura, T. (2019) · International Journal of Remote Sensing · doi:10.1080/01431161.2019.1624858 · Cited by 171
“Such model may be obtained by the model compression methods (Cheng et al. 2018) such as knowledge distillation (Hinton, Vinyals, and Dean 2014) and pruning (Han, Mao, and Dally 2016).”
MethodMulti-Channel Deep Networks for Block-Based Image Compressive Sensing
Zhou, S., Yan, H., Liu, Y., Li, C., & Zhang, J. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3014561 · Cited by 160
“The interested readers can refer to [37].”
Machine Learning in Real-Time Internet of Things (IoT) Systems: A Survey
Bian, J., Arafat, A. A., Xiong, H., Li, J., Li, L., Chen, H., Wang, J., Dou, D., & Guo, Z. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3161050 · Cited by 202
“…tasks given the vast compute and time resources required to develop neural network models on these problems, where the pre-trained model then can be used to transfer to other problems without training from the scratch which can be regarded as a form of model compression [156], [180], [181].”
BackgroundMethodProcess Monitoring, Diagnosis and Control of Additive Manufacturing
Fang, Q., Xiong, G., Zhou, M., Tamir, T. S., Yan, C.-B., Wu, H., Shen, Z., & Wang, F. (2022) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2022.3215258 · Cited by 175
“From a software aspect, some efficient neural network architectures [258] and model compression methods [259] are expected to make defect detection more efficient.”
MethodEnable Deep Learning on Mobile Devices: Methods, Systems, and Applications
Cai, H., Lin, J., Lin, Y., Liu, Z., Tang, H., Wang, H., Zhu, L., & Han, S. (2022) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3486618 · Cited by 197
“There are many existing surveys related to model compression [50, 54, 68], automated machine learning [78, 119, 317], efficient hardware architecture design [272], and optimizations for specific tasks [283].”
BackgroundTang, Y., Teng, Q., Zhang, L., Min, F., & He, J. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3015521 · Cited by 143
“For model compression [9]–[12], the existing works mainly focus on”
A Survey on Large-Scale Machine Learning
Wang, M., Fu, W., He, X., Hao, S., & Wu, X. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3015777 · Cited by 219
IoT and Big Data Applications in Smart Cities: Recent Advances, Challenges, and Critical Issues
Talebkhah, M., Sali, A. B., Marjani, M., Gordan, M., Hashim, S. J., & Rokhani, F. Z. (2021) · IEEE Access · doi:10.1109/access.2021.3070905 · Cited by 174
“The resource restrictions of edge servers and mobile devices have led to development of lightweight BDA scheme and compressing BDA model whose hardware architecture, optimization, data compression, distributed computing, and machine learning require further investigations [103].”
BackgroundXu, Q., Chen, Z., Wu, K., Wang, C., Wu, M., & Li, X. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2021.3057030 · Cited by 135
Data-Free Network Quantization With Adversarial Knowledge Distillation
Choi, Y., Choi, J., El-Khamy, M., & Lee, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00363 · Cited by 135
“The benefit of over-parameterization is empirically shown to be the key factor of the great success of deep learning, but once we find a well-trained high-accuracy model, its deployment on various inference platforms faces different requirements and challenges [3, 4].”
I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences
Oliynyk, D., Mayer, R. L., & Rauber, A. (2023) · ACM Computing Surveys · doi:10.1145/3595292 · Cited by 258
“Knowledge Distillation [28] is a model compression [29] method that allows to train a smaller version (student network) of an already trained larger (teacher) network, without decreasing accuracy.”
MethodBig Data Analytics for Large-scale Wireless Networks: Challenges and Opportunities
Dai, H.-N., Wong, R. C.-W., Wang, H., Zheng, Z., & Vasilakos, A. V. (2020) · ACM Computing Surveys · doi:10.1145/3337065 · Cited by 124
“g lightweight BDA schemes and compressing BDA models [251]. It requires the efforts in hardware design, optimization, data compression, distributed computing and machine learning to achieve this goal [252]. IX. CONCLUSION In this paper, we present a detailed survey on big data analytics (BDA) for large scale wireless…”
MethodDistilling Knowledge From a Deep Pose Regressor Network
Saputra, M. R. U., Gusmao, P., Almalioglu, Y., Markham, A., & Trigoni, N. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00035 · Cited by 117
“-1 or 1), binary networks can dramatically reduce both computation time and memory consumption at the cost of significant reduction in accuracy [5].”
BackgroundMeta Graph Transformer: A Novel Framework for Spatial–Temporal Traffic Prediction
Ye, X., Fang, S., Sun, F., Zhang, C., & Xiang, S. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.12.033 · Cited by 138
Tensaurus: A Versatile Accelerator for Mixed Sparse-Dense Tensor Computations
Srivastava, N., Jin, H., Smith, S., Rong, H., Albonesi, D. H., & Zhang, Z. (2020) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00062 · Cited by 159
“Recently, tensor factorizations have also achieved promising results in compressing deep neural networks [8–10].”
BackgroundDeep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04311 · Cited by 122
“architectures on such equipment may be computationally unfeasible, unless appropriate model tuning is performed. To address this issue, ongoing research improves existing deep learning architectures [509], such that the inference process does not violate latency or energy constraints [510], [511], nor raise any privac…”
BackgroundTransformed ℓ1 regularization for learning sparse deep neural networks
Ma, R., Miao, J., Niu, L., & Zhang, P. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.08.015 · Cited by 150
DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
Wiedemann, S., Kirchhoffer, H., Matlage, S., Haase, P., Marban, A., Marinc, T., Neumann, D., Nguyen, T., Schwarz, H., Wiegand, T., et al. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2969554 · Cited by 113
“There has been a plethora of work focusing on the topic of neural network compression [9], [10].”
BackgroundDave, S., Baghdadi, R., Nowatzki, T., Avancha, S., Shrivastava, A., & Li, B. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3098483 · Cited by 111
“Compact models: Cheng et al. [265] surveyed techniques for parameter pruning and low-rank factorization.”
MethodCompressing deep neural networks on FPGAs to binary and ternary precision with hls4ml
Ngadiuba, J., Lončar, V., Pierini, M., Summers, S., Guglielmo, G. D., Duarte, J., Harris, P. C., Rankin, D. S., Jindariani, S., Liu, M., et al. (2020) · Machine Learning Science and Technology · doi:10.1088/2632-2153/aba042 · Cited by 152
“Comprehensive reviews of these techniques can be found in reference [10, 11].”
BackgroundMedical Image Segmentation With Limited Supervision: A Review of Deep Network Models
Peng, J., & Wang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3062380 · Cited by 132
“employed: model compression [322] and efficient model architecture designs [2], [320], [323].”
MethodA recurrent convolutional neural network approach for sensorless force estimation in robotic surgery
Marbán, A., Srinivasan, V., Samek, W., Fernández, J., & Casals, A. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.01.011 · Cited by 108
Machine Learning in Manufacturing towards Industry 4.0: From ‘For Now’ to ‘Four-Know’
Chen, T., Sampath, V., May, M. C., Shan, S., Jorg, O. J., Martín, J. J. A., Stamer, F., Fantoni, G., Tosello, G., & Calaon, M. (2023) · Applied Sciences · doi:10.3390/app13031903 · Cited by 131
“In particular, obtaining high accuracy with a neural network requires on millions or even billions of parameters [170].”
BackgroundGeneration of High-Resolution Handwritten Digits with an Ion-Trap Quantum Computer
Rudolph, M. S., Toussaint, N. B., Katabarwa, A., Johri, S., Peropadre, B., & Perdomo-Ortiz, A. (2022) · Physical Review X · doi:10.1103/physrevx.12.031010 · Cited by 94
“[3] Y.”
BackgroundGeneration of High-Resolution Handwritten Digits with an Ion-Trap Quantum Computer
Rudolph, M. S., Toussaint, N. B., Katabarwa, A., Johri, S., Peropadre, B., & Perdomo-Ortiz, A. (2020) · arXiv · doi:10.48550/arxiv.2012.03924 · Cited by 97
“[3] Y.”
BackgroundFederated Multi-Task Learning for Joint Diagnosis of Multiple Mental Disorders on MRI Scans
Huang, Z.-A., Hu, Y. F., Liu, R., Xue, X., Zhu, Z., Song, L., & Tan, K. C. (2022) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2022.3210940 · Cited by 111
“Following the steps of minimizing (cid:5) 2 objective in FitNet [22], we can define the distance measure of intermediate feature maps between the m th source task S m ( x ) and the n th target task where r θ is used as a pointwise convolution with a linear transformation controlled by θ [23].”
MethodA Knee-Guided Evolutionary Algorithm for Compressing Deep Neural Networks
Zhou, Y., Yen, G. G., & Zhang, Y. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2928174 · Cited by 101
“However, KD is limited to be applied to classification tasks with a softmax loss function, which hampers its usage in practical applicability [7].”
BackgroundInfluentialUniversal Deep Neural Network Compression
Choi, Y., El‐Khamy, M., & Lee, J. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2975903 · Cited by 111
“, see [4], [5]).”
Edge Intelligence: A Review of Deep Neural Network Inference in Resource-Limited Environments
Ngo, D., Park, H.-C., & Kang, B. (2025) · Electronics · doi:10.3390/electronics14122495 · Cited by 158
Tuttle, J. F., Blackburn, L. D., Andersson, K., & Powell, K. M. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.116886 · Cited by 99
Edge AI for Internet of Energy: Challenges and perspectives
Himeur, Y., Sayed, A. N., Alsalemi, A., Bensaali, F., & Amira, A. (2023) · Internet of Things · doi:10.1016/j.iot.2023.101035 · Cited by 132
Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression ApproachPreprint
Zhou, W., Veitch, V., Austern, M., Adams, R. P., & Orbanz, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.05862 · Cited by 199
Image analysis-based recognition and quantification of grain number per panicle in rice
Wu, W., Liu, T., Zhou, P., Yang, T., Li, C., Zhong, X., Sun, C., Liu, S., & Guo, W. (2019) · Plant Methods · doi:10.1186/s13007-019-0510-0 · Cited by 100
Self-citationIntelligent Radio Signal Processing: A Survey
Pham, Q., Nguyen, N. T., Huynh‐The, T., Le, L. B., Lee, K., & Hwang, W. (2021) · IEEE Access · doi:10.1109/access.2021.3087136 · Cited by 99
“Compressing and accelerating DL techniques can be classified into four main groups [182]: 1) parameter pruning and quantization, 2) low-rank factorization, 3) transferred/compact convolutional filters, and 4) knowledge distil-”
MethodA Compact Convolutional Neural Network for Surface Defect Inspection
Huang, Y., Qiu, C., Wang, X., Wang, S., & Yuan, K. (2020) · Sensors · doi:10.3390/s20071974 · Cited by 85
“More practical approaches and suggestions can be seen in Reference [40].”
BackgroundDeep learning methods for solving linear inverse problems: Research directions and paradigms
Bai, Y., Chen, W., Chen, J., & Guo, W. (2020) · Signal Processing · doi:10.1016/j.sigpro.2020.107729 · Cited by 103
A Survey on Deep-Learning-Based Real-Time SAR Ship Detection
Li, J., Chen, J., Cheng, P., Yu, Z., Yu, L., & Chi, C. (2023) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2023.3244616 · Cited by 98
Evolutionary Compression of Deep Neural Networks for Biomedical Image Segmentation
Zhou, Y., Yen, G. G., & Zhang, Y. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2933879 · Cited by 80
“CNN compression [24] aims to improve the efficiency of CNNs by compressing its parameters.”
BackgroundInfluentialDeep Model Compression and Architecture Optimization for Embedded Systems: A Survey
Berthelier, A., Château, T., Duffner, S., García, C., & Blanc, C. (2020) · Journal of Signal Processing Systems · doi:10.1007/s11265-020-01596-1 · Cited by 98
Teacher-student collaborative knowledge distillation for image classification
Xu, C., Gao, W., Li, T., Bai, N., Li, G., & Zhang, Y. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03486-4 · Cited by 79
Hamrani, A., Agarwal, A. K., Allouhi, A., & McDaniel, D. (2023) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02171-8 · Cited by 89
“Although deep learning is frequently compared to classical ML algorithms (such as support vector machine (SVM) and random forest (RF)), it is generally found to be superior in terms of prediction accuracy and data adaptability (Hamrani et al., 2020; Wang et al., 2018).”
RACE-Net: A Recurrent Neural Network for Biomedical Image Segmentation
Chakravarty, A., & Sivaswamy, J. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2852635 · Cited by 76
“…on small training datasets which is particularly useful in the medical domain where obtaining the ground truth markings is expensive; b) It reduces the computation time and memory requirements of the pre-trained architecture allowing it to be deployed on systems with limited resources [18].”
BackgroundBeyond human-level accuracy: computational challenges in deep learning
Hestness, J., Ardalani, N., & Diamos, G. (2019) · Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3293883.3295710 · Cited by 70
“ts, but some algorithmic optimizations may be promising to chip away at this gap. Model compression or distillation, and low-precision or sparse computation may reduce model or activation tensor size [9], and reduce memory requirements by 1:5–10. Currently, many challenges exist to use these techniques during model tr…”
BackgroundPixelhop++: A Small Successive-Subspace-Learning-Based (Ssl-Based) Model For Image Classification
Chen, Y., Rouhsedaghat, M., You, S. O., Rao, R., & Kuo, C. ‐. J. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9191012 · Cited by 94
“topic in recent years since small models are essential to mobile and edge computing applications. There has been a lot of research dedicated to neural network model compression and acceleration, e.g. [1, 2, 3, 4, 5]. Techniques such as parameter pruning, quantization, binarization and sharing, low-rank factorization,…”
MethodEnergy-Efficient Processing and Robust Wireless Cooperative Transmission for Edge Inference
Yang, K., Shi, Y., Yu, W., & Ding, Z. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2979523 · Cited by 65
“In addition to the low-latency, improving the energy efficiency [5] is also critical due to the high computational complexity of processing DNNs, for which a number of works focusing on model compression methods [6], [7].”
Jin, P., Lu, L., Tang, Y., & Karniadakis, G. E. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.06.024 · Cited by 79
“There are also other approaches, such as compression (Arora et al., 2018; Baykal et al., 2018; Zhou et al., 2018; Cheng et al., 2018), Fourier analysis (Rahaman et al., 2018; Xu et al., 2019), “double descent” risk curve (Belkin et al., 2018), PAC-Bayesian framework (Neyshabur et al., 2017b;…”
Bayesian-based uncertainty-aware tool-wear prediction model in end-milling process of titanium alloy
Kim, G., Yang, S. M., Kim, D. M., Kim, S., Choi, J. G., Ku, M., Lim, S., & Park, H. W. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110922 · Cited by 73
Continual Learning With Knowledge Distillation: A Survey
Li, S., Su, T., Zhang, X.-Y., & Wang, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3476068 · Cited by 97
“Initially, the main application of KD is model compression [33], [34], addressing the increased computational and storage demands due to the growth in deep learning model size, which limits suitability for real-time processing and deployment on resource-constrained systems.”
Human-grounded Evaluations of Explanation Methods for Text Classification
Lertvittayakumjorn, P., & Toni, F. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1523 · Cited by 60
“, measuring model complexity (Bianchini and Scarselli, 2014) and model compression (Cheng et al., 2018).”
BackgroundTuttle, J. F., Blackburn, L. D., & Powell, K. M. (2020) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2020.106990 · Cited by 65
Big Data Analytics for Large Scale Wireless Networks: Challenges and Opportunities
Dai, H., Wong, R. C.-W., Wang, H. H., Zheng, Z., & Vasilakos, A. V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.08069 · Cited by 59
“g lightweight BDA schemes and compressing BDA models [251]. It requires the efforts in hardware design, optimization, data compression, distributed computing and machine learning to achieve this goal [252]. IX. CONCLUSION In this paper, we present a detailed survey on big data analytics (BDA) for large scale wireless…”
MethodFreire, P. J., Napoli, A., Spinnler, B., Anderson, M., Ron, D. A., Schairer, W., Bex, T., Costa, N. M. S., Turitsyn, S. K., & Prilepsky, J. E. (2023) · Journal of Lightwave Technology · doi:10.1109/jlt.2023.3234327 · Cited by 80
Recent Advances on Neural Network Pruning at Initialization
Wang, H., Qin, C., Bai, Y., Zhang, Y., & Fu, Y. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/786 · Cited by 114
Compacting Deep Neural Networks for Internet of Things: Methods and Applications
Zhang, K., Ying, H., Dai, H., Li, L., Peng, Y., Guo, K., & Yu, H. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3063497 · Cited by 69
“The work [4] presents an overview of model compression and acceleration of DNNs while it only considers Convolu-tional Neural Networks (CNNs) and does not consider other types DNNs, such as Recurrent Neural Networks (RNNs).”
Advancements in Accelerating Deep Neural Network Inference on AIoT Devices: A Survey
Cheng, L., Gu, Y., Liu, Q., Yang, L., Liu, C., & Wang, Y. (2024) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2024.3353176 · Cited by 64
Ps and Qs: Quantization-Aware Pruning for Efficient Low Latency Neural Network Inference
Hawks, B., Duarte, J. M., Fraser, N. J., Pappalardo, A., Tran, N. V., & Umuroglu, Y. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.676564 · Cited by 110
“Several approaches have been successfully deployed to compress networks (Cheng et al., 2018; Choudhary et al., 2020; Deng et al., 2020).”
BackgroundPerformance-Aware NILM Model Optimization for Edge Deployment
Sykiotis, S., Athanasoulias, S., Kaselimi, M., Doulamis, A. D., Doulamis, N. D., Stanković, L., & Stanković, V. (2023) · IEEE Transactions on Green Communications and Networking · doi:10.1109/tgcn.2023.3244278 · Cited by 58
“Despite the great success of deep learning in diverse applications, neural networks often possess a vast number of parameters, leading to significant challenges in deploying deep learning systems to a resource-limited device [33], [34].”
BackgroundOn Removing Algorithmic Priority Inversion from Mission-critical Machine Inference Pipelines
Liu, S., Yao, S., Fu, X., Tabish, R., Yu, S., Bansal, A., Yun, H., Sha, L., & Abdelzaher, T. (2020) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss49844.2020.00037 · Cited by 69
“In the broader neural network research literature, much work was done on model compression and acceleration [18]– [20].”
Background
Metadata
- Authors
- Cheng, Y., Wang, D., Zhou, P., & Zhang, T.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- Harbin Institute of Technology (CN)
- Huazhong University of Science and Technology (CN)
- Tsinghua University (CN)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2018-01
- Volume
- 35
- Issue
- 1
- Pages
- 126-136
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W2783538964
- Cited by
- 817
- References
- 67
- Citations by year
- 2026: 114 · 2025: 132 · 2024: 116 · 2023: 110 · 2022: 109 · 2021: 92 · 2020: 89 · 2019: 41 · 2018: 14
- FWCI
- 29.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- model compression, deep neural network, inference acceleration, parameter reduction, convolutional neural networks, training efficiency, GPU computing, face verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.