Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges
The DOI matches its Crossref record; 61 of the 67 references it cites were found in a registry or an index.
Abstract
In recent years, deep neural networks (DNNs) have received increased attention, have been applied to different applications, and achieved dramatic accuracy improvements in many tasks. These works rely on deep networks with millions or even billions of parameters, and the availability of graphics processing units (GPUs) with very high computation capability plays a key role in their success. For example, Krizhevsky et al. achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully connected layers. Usually, it takes two to three days to train the whole model on the ImagetNet data set with an NVIDIA K40 machine. In another example, the top face-verification results from the Labeled Faces in the Wild (LFW) data set were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally connected, and fully connected layers. It is also very time-consuming to train such a model to obtain a reasonable performance. In architectures that only rely on fully connected layers, the number of parameters can grow to billions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [26]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Recently, more state-of-the-art architectures are used as baseline models in many works, including network in networks [67], VGGNets [68], and ResNets [69].”
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“[1] achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully-connected layers.”
MethodInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The work in [51] introduced a KD compression framework, which eased the training of deep networks by following a student-teacher paradigm, in which the student was penalized according to a softened version of the teacher’s output.”
BackgroundDeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · cited 1×
“example is the top face verification results on the Labeled Faces in the Wild (LFW) dataset were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally-connected, and fully-connected layers [2], [3].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“works that directly train CNNs with binary weights, for instance, BinaryConnect [13], BinaryNet [14] and XNORNetworks [15].”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
Method
Cited by
Items 101–200 of 818
Dynamic Split Computing for Efficient Deep EDGE Intelligence
Bakhtiarnia, A., Milošević, N., Zhang, Q., Bajović, D., & Iosifidis, A. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096914 · Cited by 76
Disaster-Resilient Emergency Communication With Intelligent Air–Ground Cooperation
Tang, X., Chen, F., Wang, F., & Jia, Z. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3305549 · Cited by 53
“Even though current deep learning methods can be good at feature representation, the centralized model with high-dimensional state input would take huge amounts of time for training, and can hardly be transferred in a new environment [41].”
BackgroundAcoustic scene classification: A comprehensive survey
Ding, B., Zhang, T., Wang, C., Liu, G., Liang, J., Hu, R., Wu, Y., & Guo, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121902 · Cited by 73
Self-citationEfficient High-Resolution Deep Learning: A Survey
Bakhtiarnia, A., Zhang, Q., & Iosifidis, A. (2024) · ACM Computing Surveys · doi:10.1145/3645107 · Cited by 86
“with other efficient deep learning methods, such as model compression [154], dynamic inference [155], collaborative inference [156] and continual inference [157], is an unexplored area of research.”
BackgroundEvolutionary Shallowing Deep Neural Networks at Block Levels
Zhou, Y., Yen, G. G., & Zhang, Y. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3059529 · Cited by 59
“Various network compression approaches have been investigated for accelerating DNNs [22].”
BackgroundAlqudah, A. M., & Moussavi, Z. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.063643 · Cited by 101
Lee, J., Mukhanov, L., Molahosseini, A. S., Minhas, U. I., Yang, H., del Rincón, J. M., Dichev, K., Hong, C.-H., & Vandierendonck, H. (2023) · ACM Computing Surveys · doi:10.1145/3587095 · Cited by 70
“[35, 36] conducted a survey on the model compression techniques including pruning, low-rank factorization, compact convolution, and knowledge distillation.”
BackgroundButt, M. A., Qayyum, A., Ali, H. M., Al‐Fuqaha, A., & Qadir, J. (2022) · Computers & Security · doi:10.1016/j.cose.2022.103058 · Cited by 61
A Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 80
“Model compression and acceleration techniques was divided into four different approaches[35]: parameter pruning and sharing[36][37][38][39][40], low-rank factorization[41][42], transferred/compact convolutional layers[43] and knowledge distillation[44].”
MethodGlobal Aligned Structured Sparsity Learning for Efficient Image Super-Resolution
Wang, H., Zhang, Y., Qin, C., Gool, L. V., & Fu, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3268675 · Cited by 55
“Unstructured pruning, on the other hand,resultsinirregularsparsity,whichisgoodforcompression, i.e.,reducingmodelsize,butnon-triviallychallengingtoachieve practicalspeedaccelerationonhardware[13],[14],[29],[39].”
BackgroundInfluentialKnowledge distilling based model compression and feature learning in fault diagnosis
Zhang, W., Biswas, G., Zhao, Q., Zhao, H., & Feng, W. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.105958 · Cited by 49
Competent Ultra Data Compression By Enhanced Features Excerption Using Deep Learning Techniques
Nagaraj, P., Rao, J. S., Muneeswaran, V., Kumar, A. S., & Sudar, K. M. (2020) · International Conference on Intelligent Computing and Control Systems (ICICCS) · doi:10.1109/iciccs48265.2020.9121126 · Cited by 48
Deep convolutional autoencoder thermography for artwork defect detection
Liu, Y., Wang, F., Liu, K., Mostacci, M., Yao, Y., & Сфарра, С. (2023) · Quantitative InfraRed Thermography Journal · doi:10.1080/17686733.2023.2225246 · Cited by 47
“A convolutional structure has the advantage of weight sharing, which helps solve the problem of parameter redundancy in fully connected networks and accelerates model training [26].”
Choudhary, T., Gujar, S. A., Goswami, A., Mishra, V. K., & Badal, T. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03893-7 · Cited by 53
Deep Neural Network Acceleration Based on Low-Rank Approximated Channel Pruning
Chen, Z., Chen, Z., Lin, J., Liu, S., & Li, W. (2020) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2019.2958937 · Cited by 48
“There are many existing works aiming at deep neural network compression and acceleration from different perspectives, such as low-rank approximation, weight quantization, distill learning, pruning, and so on [17].”
BackgroundData and Knowledge Dual-Driven Automatic Modulation Classification for 6G Wireless Communications
Ding, R., Zhou, F., Wu, Q., Dong, C., Han, Z., & Dobre, O. A. (2023) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2023.3316197 · Cited by 52
“This drawback makes it unrealistic since the real-time performance is of crucial importance [34], especially in 6G wireless communication networks.”
BackgroundNeural Network Decoders for Large-Distance 2D Toric Codes
Ni, X. (2020) · Quantum · doi:10.22331/q-2020-08-24-310 · Cited by 35
Huang, Q. (2022) · AI · doi:10.3390/ai3010011 · Cited by 48
“According to [21,22], the common model compression approaches include the following: network pruning [23], weight factorization [24], quantization [25], and weight sharing [26].”
MethodOptimization Methods, Challenges, and Opportunities for Edge Inference: A Comprehensive Survey
Zhang, R., Jiang, H., Wang, W., & Liu, J. (2025) · Electronics · doi:10.3390/electronics14071345 · Cited by 76
Data Shuffling in Wireless Distributed Computing via Low-Rank Optimization
Yang, K., Shi, Y., & Ding, Z. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2019.2912139 · Cited by 43
“Storage size is often one of the key limiting factors in a single device when deploying deep learning model [3], [5].”
BackgroundEnabling Serverless Deployment of Large-Scale AI Workloads
Christidis, A., Moschoyiannis, S., Hsu, C., & Davies, R. (2020) · IEEE Access · doi:10.1109/access.2020.2985282 · Cited by 52
EdgeKE: An On-Demand Deep Learning IoT System for Cognitive Big Data on Industrial Edge Devices
Fang, W. W., Feng, X., Ding, Y., Xiong, N. N., & Leung, V. C. M. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3044930 · Cited by 48
“On the other hand, knowledge distillation exploits knowledge transfer to compress model by following a teacher-student paradigm [7], in which the compact model (student) obtains knowledge from the complex model (teacher) by learning the class distributions output of the teacher via softened softmax [3].”
BackgroundInfluentialSingle-layer vision transformers for more accurate early exits with less overhead
Bakhtiarnia, A., Zhang, Q., & Iosifidis, A. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.06.038 · Cited by 58
Pruning CNN filters via quantifying the importance of deep visual representations
Alqahtani, A., Xie, X., Jones, M. W., & Essa, E. (2021) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2021.103220 · Cited by 47
Diet code is healthy: simplifying programs for pre-trained models of code
Zhang, Z., Zhang, H., Shen, B., & Gu, X. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549094 · Cited by 41
“We will also investigate model compression [8, 9] and distillation [39] techniques to further reduce the size of pre-trained programming language models.”
MethodSamanta, R. K., Sadhukhan, B., Samaddar, H., Sarkar, S., Koner, C., & Ghosh, M. (2022) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12114 · Cited by 53
Noothout, J. M. H., Leßmann, N., van Eede, M. C., van Harten, L. D., Sogancioglu, E., Heslinga, F. G., Veta, M., van Ginneken, B., & Išgum, I. (2022) · Journal of medical imaging · doi:10.1117/1.jmi.9.5.052407 · Cited by 55
“11 Techniques that reduce the footprint of CNN ensembles and other large and complex models are known as model compression techniques.”
BackgroundData-free knowledge distillation in neural networks for regression
Kang, M., & Kang, S. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114813 · Cited by 47
Deep neural network compression through interpretability-based filter pruning
Yao, K., Cao, F., Leung, Y., & Liang, J. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108056 · Cited by 47
“Filter pruning has become an important strategy to compress and accelerate DNNs [9] .”
BackgroundDeep Learning for Mobile Mental Health: Challenges and recent advances
Han, J., Zhang, Z., Mascolo, C., André, E., Tao, J., Zhao, Z., & Schuller, B. W. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3099293 · Cited by 49
“Such technologies include compact network design, parameter sharing and pruning, low-rank decomposition, model quantisation, and knowledge distillation, and these solutions can be implemented individually or collaboratively [26].”
BackgroundKim, S., Kwak, S., & Ko, B. C. (2019) · IEEE Access · doi:10.1109/access.2019.2892425 · Cited by 41
“It is known that among these four categories, the performance of the teacher-student framework matches or is superior to that of the teacher’s framework and requires considerably fewer parameters and multiplications [24].”
BackgroundCococcioni, M., Rossi, F., Ruffaldi, E., Saponara, S., & de Dinechin, B. D. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2988436 · Cited by 49
“Hence, onboard computing units for DNNs should be optimized in terms of the ratio between the signal processing throughput performance and resources (memory, bandwidth, power consumption, and so on) [15]–[17].”
BackgroundiToF2dToF: A Robust and Flexible Representation for Data-Driven Time-of-Flight Imaging
Gutierrez-Barragan, F., Chen, H., Gupta, M., Velten, A. U., & Gu, J. (2021) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2021.3126533 · Cited by 44
“edu/∼felipe/project-pages/2021-itof2dtof/ weight neural networks [5], or through model compression and inference acceleration [9], [10].”
BackgroundKody, A., Chevalier, S. C., Chatzivasileiadis, S., & Molzahn, D. K. (2022) · Electric Power Systems Research · doi:10.1016/j.epsr.2022.108282 · Cited by 39
Literature Review of Deep Network Compression
Alqahtani, A., Xie, X., & Jones, M. W. (2021) · Informatics · doi:10.3390/informatics8040077 · Cited by 50
Structured Sparsity Learning for Efficient Video Super-Resolution
Xia, B., He, J., Zhang, Y., Wang, Y., Tian, Y., Yang, W., & Gool, L. V. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02168 · Cited by 57
“Network pruning [4, 5, 34, 36, 38] is widely used to remove a set of redundant parameters for network acceleration.”
Explainable AI in Healthcare Application
Sindiramutty, S. R., Tee, W. J., Balakrishnan, S., Kaur, S., Thangaveloo, R., Jazri, H., Khan, N. A., Gharib, A. H., & Manchuri, A. R. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-6684-6361-1.ch005 · Cited by 45
Artificial Intelligence as a Service (AIaaS) for Cloud, Fog and the Edge: State-of-the-Art Practices
Syed, N., Anwar, A., Baig, Z. A., & Zeadally, S. (2025) · ACM Computing Surveys · doi:10.1145/3712016 · Cited by 60
Optimizing Edge Computing Performance using Machine Learning for Low-Latency Applications
Karthiga, M., Paramasivam, J., Raja, P. V., Suman, S. K., Bhagyalakshmi, L., & Poddar, H. (2025) · World Skills Conference on Universal Data Analytics and Sciences (WorldSUAS) · doi:10.1109/worldsuas66815.2025.11198929 · Cited by 38
“[6].”
Deep Learning-Based Cancer Detection-Recent Developments, Trend and Challenges
Kumar, G., & Alqahtani, H. (2021) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2022.018418 · Cited by 45
Survey: Exploiting Data Redundancy for Optimization of Deep Learning
Chen, J.-A., Niu, W., Ren, B., Wang, Y., & Shen, X. (2022) · ACM Computing Surveys · doi:10.1145/3564663 · Cited by 48
Edge intelligence secure frameworks: Current state and future challenges
Villar-Rodríguez, E., Arostegi, M., Torre-Bastida, A. I., Regueiro, C., & López-de-Armentia, J. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103278 · Cited by 53
Luo, H., Chen, T., Li, X., Li, S., Zhang, C., Zhao, G., & Liu, X. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3157957 · Cited by 43
“Besides, knowledge distillation has become an important branch in the field of DNN compression and acceleration [13], [21].”
BackgroundImproving End-to-End Single-Channel Multi-Talker Speech Recognition
Zhang, W., Chang, X., Qian, Y., & Watanabe, S. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2988423 · Cited by 47
“distillation for model compression in most previous works [46]– [48], we propose a knowledge distillation framework [32] to transfer knowledge from a single-speaker model, i.”
BackgroundWhen classification accuracy is not enough: Explaining news credibility assessment
Przybyła, P., & Soto, A. J. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102653 · Cited by 52
Sepahvand, M., Abdali-Mohammadi, F., & Taherkordi, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117474 · Cited by 45
Wisanwanichthan, T., & Thammawichai, M. (2025) · Computers · doi:10.3390/computers14070291 · Cited by 60
Wang, Z., Du, L., & Li, Y. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3104267 · Cited by 39
Zheng, Y., Sun, P., Ren, Q., Xu, W., & Zhu, D. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127124 · Cited by 38
Lightweight Deep Learning Models For Edge Devices—A Survey
Musa, A., Kakudi, H. A., Hassan, M., Hamada, M., Umar, U., & Salisu, M. L. (2025) · International Journal of Computer Information Systems and Industrial Management Applications · doi:10.70917/2025014 · Cited by 75
Yu-feng, B., & Qi, L. (2021) · Mobile Information Systems · doi:10.1155/2021/3532963 · Cited by 35
“In this way, the sample information is transferred layer by layer from the first input layer, and finally, a predicted value will be obtained [12].”
BackgroundSanagala, S. S., Agarwal, M., Utkarsh, K., Gupta, S. K., Koppula, V. K., & Suri, J. S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07567-w · Cited by 36
“[51] described another four model compression techniques: (1) model pruning using quantization and binarization, (2) low-rank matrix factorization and filtering to speed up matrix multiplication, (3) knowledge distillation and transfer learning, and 4) hybrid models combining any two of the other models.”
MethodMultigrid-Augmented Deep Learning Preconditioners for the Helmholtz Equation
Azulay, Y., & Treister, E. (2022) · SIAM Journal on Scientific Computing · doi:10.1137/21m1433514 · Cited by 52
EvoQ: Mixed Precision Quantization of DNNs via Sensitivity Guided Evolutionary Search
Yuan, Y., Chen, C., Hu, X., & Peng, S. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207413 · Cited by 45
“computational complexity hinders the deployment of DNNs on embedded devices [1] [2].”
BackgroundA tucker decomposition based knowledge distillation for intelligent edge applications
Dai, C., Liu, X., Li, Z., & Chen, M. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.107051 · Cited by 35
Optimizing the Deep Neural Networks by Layer-Wise Refined Pruning and the Acceleration on FPGA
Li, H., Yue, X., Wang, Z., Chai, Z., Wang, W., Tomiyama, H., & Meng, L. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/8039281 · Cited by 35
“…DNNs and have achieved significant progress. solutions of software level can be briefly classified into four categories according to their properties [8]: pruning and quantization which reduce the computations to accelerate the network [9–11], low-rank factorization which utilizes the matrix/tensor…”
BackgroundFedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantification
Hui, C. H. E. N., Liu, H., Wu, Z., Fan, X., & Cao, L. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3562164 · Cited by 71
Hendy, H., & Merkel, C. E. (2022) · Journal of Electronic Imaging · doi:10.1117/1.jei.31.1.010901 · Cited by 38
“8 The second approach, which is the topic of this review, is to codesign hardware and algorithms that more closely emulate the anatomical and physiological characteristics that underlie the incredible efficiency of biological intelligence.”
BackgroundSepahvand, M., & Abdali-Mohammadi, F. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106476 · Cited by 32
Real-Time Semantic Segmentation: A brief survey and comparative study in remote sensing
Broni-Bediako, C., Xia, J., & Yokoya, N. (2023) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2023.3321258 · Cited by 34
“Various methods adopted in the literature [7, 111, 112] include singular value decomposition (SVD) [113], and tensor decomposition methods such as Tucker decomposition [114, 115] and canonical polyadic (CP) decomposition [116].”
MethodWang, G., Zhang, N., Wang, J., Liu, W., Xie, Y., & Chen, H. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3354944 · Cited by 35
Avci, M. B., Kurul, F., Topkaya, S. N., & Çetin, A. E. (2025) · Microchimica Acta · doi:10.1007/s00604-025-07523-0 · Cited by 38
Joint Progressive Knowledge Distillation and Unsupervised Domain Adaptation
Nguyen-Meidine, L. T., Granger, É., Kiran, M., Dolz, J., & Blais-Morin, L.-A. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206989 · Cited by 33
“For a comprehensive survey on compression techniques for CNNs, we refer the reader to [32].”
BackgroundCut, Distil and Encode (CDE): Split Cloud-Edge Deep Inference
Sbai, M., Saputra, M. R. U., Trigoni, N., & Markham, A. (2021) · IEEE International Conference on Sensing, Communication, and Networking (SECON) · doi:10.1109/secon52354.2021.9491600 · Cited by 36
Enhancing Edge Computing with Machine Learning for Low-Latency Applications
Dhumpati, R., Kasireddy, L. C., Gunavardini, V., Suman, S. K., Bhagyalakshmi, L., & Talwar, S. (2025) · IEEE International Conference on Advances in Computing Research on Science Engineering and Technology (ACROSET) · doi:10.1109/acroset66531.2025.11280761 · Cited by 38
“These include model compression methods like pruning, quantization, and knowledge distillation, which reduce model size and computational requirements [8-9].”
DNN compression by ADMM-based joint pruning
Lee, G., & Lee, K. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107988 · Cited by 30
Cnns in land cover mapping with remote sensing imagery: a review and meta-analysis
Kotaridis, I., & Lazaridou, M. A. (2023) · International Journal of Remote Sensing · doi:10.1080/01431161.2023.2255354 · Cited by 32
“The success of CNNs comes at the cost of increased computational requirements (Alzubaidi et al. 2021; Cheng et al. 2018; Mahdianpari et al. 2018).”
BackgroundHohman, F., Kery, M. B., Ren, D., & Moritz, D. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642109 · Cited by 50
“The literature on efficient machine learning is broader than the scope of this paper, and for a comprehensive look we refer readers to the excellent survey papers cited here and below [17, 19, 22, 68, 101].”
BackgroundBoukerche, A., & Sha, M. (2021) · ACM Computing Surveys · doi:10.1145/3460770 · Cited by 27
“Parameter pruning is another approach to compress models, which removes the unnecessary network connections by setting a pre-defined threshold, such as [18, 58].”
BackgroundSepahvand, M., Abdali-Mohammadi, F., & Taherkordi, A. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105560 · Cited by 32
Lightweight network learning with Zero-Shot Neural Architecture Search for UAV images
Yao, F., Wang, S., Ding, L., Zhong, G., Bullock, L. B., Xu, Z., & Dong, J. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110142 · Cited by 26
Khan, N. M., Khan, U. A., Asif, M., & Zafar, M. H. (2024) · Energy Reports · doi:10.1016/j.egyr.2024.04.035 · Cited by 31
Learning Sparse Low-Precision Neural Networks With Learnable Regularization
Choi, Y., El‐Khamy, M., & Lee, J. (2020) · IEEE Access · doi:10.1109/access.2020.2996936 · Cited by 70
“Network compression for efficient inference is of great interest for deployment of large-size DNNs on resource-limited platforms such as battery-powered mobile devices [2], [3].”
BackgroundFrom Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26
“For example, Cheng et al. [6], Mishra et al. [7], Neill [8], and Alqahtani et al. [9] mainly focused on compression algorithms of existing models, including quantization and KD.”
B 2 SFL: A Bi-Level Blockchained Architecture for Secure Federated Learning-Based Traffic Prediction
Guo, H., Meese, C., Li, W., Shen, C.-C., & Nejad, M. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3318990 · Cited by 26
Compressing 3DCNNs based on tensor train decomposition
Wang, D., Zhao, G., Li, G., Deng, L., & Wu, Y. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.07.028 · Cited by 33
“onal 3DCNN architectures, particularly the longterm temporal convolutions with large sized 3D convolutional kernels [15] and the very deep 3D ResNets [16], make the situation of inflated sizes of DNNs [19] more serious. Even worse, to the best of our knowledge, there does not exist any practice to compress 3DCNNs to sa…”
BackgroundMethodSelf-citationTraining Noise-Resilient Recurrent Photonic Networks for Financial Time Series Analysis
Passalis, N., Kirtas, M., Mourgias-Alexandris, G., Dabos, G., Pleros, N., & Tefas, A. (2020) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco47968.2020.9287649 · Cited by 33
“which increases the cost of adopting DL, as well as restricts its potential applications [2], [3].”
BackgroundDNN Model Compression for IoT Domain-Specific Hardware Accelerators
Russo, E., Palesi, M., Monteleone, S., Patti, D., Mineo, A., Ascia, G., & Catania, V. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3111723 · Cited by 27
Designing efficient convolutional neural network structure: A survey
Mi, J., Feng, J., & Huang, K.-Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.08.158 · Cited by 35
STKD: Distilling Knowledge From Synchronous Teaching for Efficient Model Compression
Su, T., Zhang, J., Yu, Z., Wang, G., & Liu, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3164264 · Cited by 34
“Hence, KD is proposed to train a compact network while retaining high accuracy [27]–[29].”
BackgroundImplementation of DNNs on IoT devices
Zhang, Z., & Kouzani, A. Z. (2019) · Neural Computing and Applications · doi:10.1007/s00521-019-04550-w · Cited by 33
Wang, L., Zhang, J., Tian, J., Li, J., Li, Z., & Tian, Q. C. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3260883 · Cited by 34
“Model compression techniques mainly include pruning, quantization, and knowledge distillation [33].”
BackgroundClownfish: Edge and Cloud Symbiosis for Video Stream Analytics
Nigade, V., Wang, L., & Bal, H. E. (2020) · IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec50012.2020.00012 · Cited by 35
“of the optimization method and the model complexity [28] and typically requires tedious parameter tuning in a large parameter space.”
MethodAutomated detection of schizophrenia using deep learning: a review for the last decade
Sharma, M. D., Patel, R. K., Garg, A., SanTan, R., & Acharya, U. R. (2023) · Physiological Measurement · doi:10.1088/1361-6579/acb24d · Cited by 32
“The vanishing gradient problem emerges when employing backpropagation and gradient-based learning approaches with artificial neural networks, especially during training (Cheng et al 2018,Mehta and Pandit 2018,Wiedemann et al 2020).”
MethodLi, W., Wang, Z., Mai, R., Ren, P., Zhang, Q., Zhou, Y., Xu, N., Zhuang, J., Xin, B., Gao, L., et al. (2023) · Visual Intelligence · doi:10.1007/s44267-023-00006-x · Cited by 32
Learning Low Precision Deep Neural Networks through Regularization.Preprint
Choi, Y., El‐Khamy, M., & Lee, J. (2018) · arXiv (Cornell University) · Cited by 29
Knowledge Enhanced Pretrained Language Models: A Compreshensive SurveyPreprint
Wei, X., Wang, S., Zhang, D., Bhatia, P., & Arnold, A. O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08455 · Cited by 72
“Another direction worth exploring is to employ model compression techniques [10] on both knowledge integration parameters (e.g., entity memory) and language model parameters of KE-PLMs.”
BackgroundChen, F., Li, Y., Chen, Y., Bian, Z., Duo, L., Zhou, Q., Zhang, L., & ADVANCED Working Group (2025) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70075 · Cited by 34
SGA-Seq: Station-Aware Graph Attention Sequence Network for Cellular Traffic Prediction
Yang, S., Wu, Q., Huang, Z., & Zhuo, Z. (2026) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2026.3664401 · Cited by 23
Biological image analysis using deep learning-based methods: Literature review
Wang, H., Shang, S., Long, L., Hu, R., Wu, Y., Chen, N., Zhang, S., Cong, F., & Lin, S. (2018) · Digital Medicine · doi:10.4103/digm.digm_16_18 · Cited by 21
“In recent years, owing to the improvement of computer hardware,[34,35] the growth of training data, and the INTRODUCTION”
BackgroundClass-specific early exit design methodology for convolutional neural networks
Bonato, V., & Bouganis, C.-S. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107316 · Cited by 28
RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Merging
Yvinec, E., Dapogny, A., Cord, M., & Bailly, K. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3179616 · Cited by 38
Personalized Knowledge Distillation for Recommender System
Kang, S., Lee, D., Kweon, W., & Yu, H. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107958 · Cited by 29
Inference-aware convolutional neural network pruning
Choudhary, T., Mishra, V. K., Goswami, A., & Jagannathan, S. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.04.031 · Cited by 25
Surface Defect Detection with Modified Real-Time Detector YOLOv3
Wang, Z., Zhu, H., Jia, X., Bao, Y., & Wang, C. (2022) · Journal of Sensors · doi:10.1155/2022/8668149 · Cited by 31
“There are several strategies to optimize deep learning nets, such as weight quantization [22], branch pruning [23, 24], and model compression [25].”
MethodLi, W., Li, L., & Yang, H. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105774 · Cited by 23
Sensitivity pruner: Filter-Level compression algorithm for deep neural networks
Guo, S., Lai, B., Yang, S., Zhao, J., & Shen, F. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109508 · Cited by 26
Greedy Shapley Client Selection for Communication-Efficient Federated Learning
Singhal, P., Pandey, S. R., & Popovski, P. (2024) · IEEE Networking Letters · doi:10.1109/lnet.2024.3363620 · Cited by 29
“…works focus on either of the two approaches: (1) model compression through techniques such as quantization, pruning, or distillation [2], [3], [4], where client updates are compressed to reduce the bandwidth usage in each round of communication with the PS, and (2) reduction of the total…”
BackgroundWielgosz, M., Mertik, M., Skoczeń, A., & Matteis, E. D. (2018) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2018.06.012 · Cited by 24
Edge Deployment of Deep Networks for Visual Detection: A Review
Zhai, H., Du, J., Ai, Y., & Hu, T. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3502539 · Cited by 35
“Model pruning can significantly reduce the model size and computational load without compromising or only a slight decrease in accuracy, thus improving the efficiency and usability of the model [95], [96], [97].”
Metadata
- Authors
- Cheng, Y., Wang, D., Zhou, P., & Zhang, T.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- Harbin Institute of Technology (CN)
- Huazhong University of Science and Technology (CN)
- Tsinghua University (CN)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2018-01
- Volume
- 35
- Issue
- 1
- Pages
- 126-136
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W2783538964
- Cited by
- 817
- References
- 67
- Citations by year
- 2026: 114 · 2025: 132 · 2024: 116 · 2023: 110 · 2022: 109 · 2021: 92 · 2020: 89 · 2019: 41 · 2018: 14
- FWCI
- 29.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- model compression, deep neural network, inference acceleration, parameter reduction, convolutional neural networks, training efficiency, GPU computing, face verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.