Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges
The DOI matches its Crossref record; 61 of the 67 references it cites were found in a registry or an index.
Abstract
In recent years, deep neural networks (DNNs) have received increased attention, have been applied to different applications, and achieved dramatic accuracy improvements in many tasks. These works rely on deep networks with millions or even billions of parameters, and the availability of graphics processing units (GPUs) with very high computation capability plays a key role in their success. For example, Krizhevsky et al. achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully connected layers. Usually, it takes two to three days to train the whole model on the ImagetNet data set with an NVIDIA K40 machine. In another example, the top face-verification results from the Labeled Faces in the Wild (LFW) data set were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally connected, and fully connected layers. It is also very time-consuming to train such a model to obtain a reasonable performance. In architectures that only rely on fully connected layers, the number of parameters can grow to billions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [26]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Recently, more state-of-the-art architectures are used as baseline models in many works, including network in networks [67], VGGNets [68], and ResNets [69].”
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“[1] achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully-connected layers.”
MethodInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The work in [51] introduced a KD compression framework, which eased the training of deep networks by following a student-teacher paradigm, in which the student was penalized according to a softened version of the teacher’s output.”
BackgroundDeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · cited 1×
“example is the top face verification results on the Labeled Faces in the Wild (LFW) dataset were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally-connected, and fully-connected layers [2], [3].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“works that directly train CNNs with binary weights, for instance, BinaryConnect [13], BinaryNet [14] and XNORNetworks [15].”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
Method
Cited by
Items 401–500 of 818
Industry 4.0 - Perspectives and Applications
Gordan, M., Ghaedi, K., & Saleh, V. (2022) · Artificial intelligence · doi:10.5772/intechopen.100660 · Cited by 16
FPGA based CNN accelerator for high-speed biomedical application
Nehete, H., Verma, G., Gupta, A., Kaushik, P., & Kaushik, B. K. (2023) · High-Speed Biomedical Imaging and Spectroscopy · doi:10.1117/12.2650101 · Cited by 6
“The two most common pooling operations in CNNs are average and max pooling.”
MethodInfluentialModel Complexity of Deep Learning: A SurveyPreprint
Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05127 · Cited by 6
A Review of Deep Learning-Enhanced Imaging Methods for Microrobots
Salehi, A., & Yu, T. (2024) · International Conference on Intelligent Robotics and Automatic Control (IRAC) · doi:10.1109/irac63143.2024.10871580 · Cited by 5
“This issue can be overcome using novel hardware-based processing solutions such as GPUs and compressing techniques, such as low-rank factorization-based techniques [59].”
InDistill: Information flow-preserving knowledge distillation for model compression
Sarridis, I., Koutlis, C., Kordopatis-Zilos, G., Kompatsiaris, I. Y., & Papadopoulos, S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00875 · Cited by 5
“The objective of Knowledge Distillation (KD), which constitutes one of the most effective ways for model compression [2, 3], is to transfer knowledge from a powerful teacher network to a smaller and faster one, in order to extend its performance capabilities.”
Crowd counting at the edge using weighted knowledge distillation
Khan, M. A., Menouar, H., Hamila, R., & Abu‐Dayya, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-90750-5 · Cited by 14
Yu, C., Xiang, S., Zhao, L., Niu, X., Liu, W., Shi, Y., Yu, L., & Hao, Y. (2025) · ACS Photonics · doi:10.1021/acsphotonics.4c02382 · Cited by 5
Deep Learning on RISC-V Platforms at the Edge: A Perspective on the Hardware and Software Support
Agosta, G., Galimberti, A., & Zoni, D. (2025) · ACM Computing Surveys · doi:10.1145/3772277 · Cited by 15
“Related surveys review compression and optimization techniques for deep neural networks (DNNs) [33], with emphasis on their suitability for hardware acceleration [42, 175], and provide taxonomies of hardware-software (HW/SW) co-design methods and accelerator-oriented optimizations for efficient…”
Velayutham, C. S., Sujit Subramanian, S., Arvindram, K., Sathya, M., Sengodan, N., Kosuri, D., Arvapalli, S. S., Thangavelu, S. S., & Jeyakumar, G. (2023) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3583133.3596333 · Cited by 5
Tanama, C., Peng, K., Marinov, Z., Stiefelhagen, R., & Roitberg, A. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342203 · Cited by 12
“From a model perspective, quantization-based model compression [13], [14], [15], [16], [17], [18], [19], [20], [21], [22] is a well-established technique for reducing the number of bits required by the model.”
MethodReal-Time Lightweight Video Super-Resolution With RRED-Based Perceptual Constraint
Wu, X., López-Tapia, S., Wang, X., Molina, R., & Katsaggelos, A. K. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3405827 · Cited by 6
“The purpose of pruning is to remove a set of redundant parameters inside deep networks so as to accelerate processing [43].”
BackgroundBDCKD: Unlocking the Power of Brownian Distance Covariance in Knowledge Distillation
Lu, G., Yin, H., Shu, Z., Wang, J., & Luo, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889695 · Cited by 7
Image classification based on self-distillation
Li, Y., Qing, L., He, X., Chen, H., & Liu, Q. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04008-y · Cited by 5
“Knowledge distillation is a common knowledge transfer method [11, 12] that allows lightweight models to learn under the guidance of heavyweight models to improve their performance [13, 14].”
BackgroundSuperFormer: Continual learning superposition method for text classification
Zeman, M., Pucer, J. F., Kononenko, I., & Bosnić, Z. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.01.040 · Cited by 8
Prototypes Sampling Mechanism for Class Incremental Learning
Tao, Z., Huang, S., & Wang, G. (2023) · IEEE Access · doi:10.1109/access.2023.3301123 · Cited by 8
“Knowledge Distillation is a widely used technique in model compression [36], [37] and transfer learning [38], which typically involves transferring knowledge from a large and complex model (often referred to as the teacher model) to a smaller and simpler model (often referred to as the student…”
MethodRen, J., Yang, S., Shi, Y., & Yang, J. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1650 · Cited by 5
Chu, T., Fang, K., Yang, J., & Huang, X. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.10.024 · Cited by 5
Comparing Different Sequences of Pruning Algorithms for Hybrid Pruning
Pragnesh, T., & Mohan, B. R. (2023) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt56998.2023.10307846 · Cited by 6
Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization
Beckers, J., van Erp, B., Zhao, Z., Kondrashov, K., & de Vries, B. (2023) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2023.3337718 · Cited by 7
An ASIC Accelerator for QNN With Variable Precision and Tunable Energy Efficiency
Wagle, A., Singh, G., Khatri, S. P., & Vrudhula, S. B. K. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3357597 · Cited by 6
“The techniques to mitigate this include maximizing the reuse of data fetched from memory [27], [28], or transferring compressed data from the memory to the processor [29], [30].”
MethodIterative algorithm computational spectrometer based on a single-hidden-layer neural network
Zheng, Y., Liao, H., Yang, L., & Chen, Y. (2024) · Optics Express · doi:10.1364/oe.524670 · Cited by 5
Optimizing Federated Learning with Heterogeneous Edge Devices
Islam, M. M., & Alawad, M. (2024) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi60790.2024.10585747 · Cited by 5
GAP: A group-based automatic pruning algorithm via convolution kernel fusion
Chen, D., Lin, K., & Deng, Q. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128488 · Cited by 5
Quantized convolutional neural networks through the lens of partial differential equations
Ben-Yair, I., Shalom, G. B., Eliasof, M., & Treister, E. (2022) · Research in the Mathematical Sciences · doi:10.1007/s40687-022-00354-y · Cited by 9
“However, this often comes at the cost of accuracy [17].”
BackgroundToward the Predictability of Dynamic Real-Time DNN Inference
Pang, W., Jiang, X., Lv, M., Gao, T. F., Liu, D., & Yi, W. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2021.3120329 · Cited by 7
Augmentation of Vegetation Index Curves Considering the Crop-Specific Phenological Characteristics
Arun, P. V., & Karnieli, A. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3142395 · Cited by 8
Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition
Rathod, J., Dawalatabad, N., Singh, S., & Gowda, D. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10582 · Cited by 16
“One of the methods to obtain the smaller models is by using model compression techniques [14, 15, 16].”
MethodChen, X., & Huang, X. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/8906385 · Cited by 5
“Related scholars have analyzed the problem of online direct sales in dual-channel supply chains under random demand and the pricing of traditional retail channels where retailers are responsible [13].”
BackgroundFactors Influencing Cost and Performance of Federated and Centralized Machine Learning
Majeed, I. A., Min, H.-K., Tadi, V. S. K., Kaushik, S., Bardhan, A., Kumaraguru, K., & Muni, R. R. D. (2022) · IEEE India Conference · doi:10.1109/indicon56171.2022.10040166 · Cited by 5
Xiao, P., Xu, T., Xiao, X., Li, W., & Wang, H. (2023) · Remote Sensing · doi:10.3390/rs15102609 · Cited by 5
Development of Light-Weight Convolutional Neural Network Model to Diagnose Tuberculosis
Raziq, A., Ahmed, N. A., Khan, S., Bizanjo, M. G., Uddin, N., & Baloch, R. (2022) · VFAST Transactions on Software Engineering · doi:10.21015/vtse.v10i3.1148 · Cited by 5
Digital Neural Network Accelerators
Rueckert, U. (2020) · The frontiers collection · doi:10.1007/978-3-030-18338-7_12 · Cited by 5
Gkalelis, N., & Mezaris, V. (2020) · IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew46912.2020.9105979 · Cited by 6
“Many approaches have been proposed to reduce the space requirements and accelerate large DCNNs, including quantization, low-rank approximations, and other [1, 2].”
BackgroundSSGD: Sparsity-Promoting Stochastic Gradient Descent Algorithm for Unbiased Dnn Pruning
Lee, C.-H., Fedorov, I., Rao, B. D., & Garudadri, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054436 · Cited by 7
“Since then, there has been a growing line of work on pruning DNNs [9, 10, 11, 12, 13, 14, 15].”
BackgroundCompression of Deep Convolutional Neural Networks under Joint Sparsity ConstraintsPreprint
Choi, Y., El‐Khamy, M., & Lee, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08303 · Cited by 8
“Thus, it is of great interest to compress large-size CNNs into compact forms to lower their storage requirements and to reduce their computational costs [2, 3].”
BackgroundDemetgül, M., & Lazarova‐Molnar, S. (2025) · Measurement · doi:10.1016/j.measurement.2025.120203 · Cited by 7
RAMS: Runtime Adaptive Memory Scaling for Tiny Deep Learning on IoT Devices
Chu, H., Xin, H., Zheng, X., Liu, L., & Ma, H. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3651722 · Cited by 5
Ghosh, A. (2026) · Universal library of engineering technology. · doi:10.70315/uloap.ulete.2026.0301002 · Cited by 7
Sensor Fault Diagnosis and Data Reconstruction in Mechanical Systems: A Comprehensive Review
Sun, R.-B., Su, Y., Yang, Z.-B., & Chen, X. (2026) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2026.3687361 · Cited by 5
“To enable more sophisticated edge intelligence, model compression techniques such as pruning, quantization, and knowledge distillation should be employed to deploy complex deep learning models on embedded devices [188].”
A Precision-Aware Neuron Engine for DNN Accelerators
Vishwakarma, S., Raut, G., Jaiswal, S., Vishvakarma, S. K., & Ghai, D. (2024) · SN Computer Science · doi:10.1007/s42979-024-02851-z · Cited by 7
“Pruning aims to eliminate the subset of network units (i.e. weights or filters) which are least important for the network’s intended task [16].”
Mutual GNN-MLP distillation for robust graph adversarial defense
Deng, B., Chen, J., Hu, Y., Chen, C., & Zhang, T. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107513 · Cited by 8
“In non-graph models, inference acceleration can be achieved through techniques such as pruning and quantization, which reduce multiplication-and-accumulation (MAC) operations [38].”
MethodSelf-citationSCOPE: Stochastic Cartographic Occupancy Prediction Engine for Uncertainty-Aware Dynamic Navigation
Xie, Z., & Dames, P. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3578234 · Cited by 12
“As summarized in surveys [28], [29], the main methods of software acceleration can be divided into three categories: 1) network pruning; 2) weight quantization and sharing; and 3) knowledge distillation.”
Li, J., & Wu, Q. (2025) · Cluster Computing · doi:10.1007/s10586-024-05032-y · Cited by 6
“This approach of weight sharing not only mitigates transmission overhead but also thwarts attackers from acquiring full data for inversion attacks, rendering it highly favored by a multitude of scholars [22 – 24].”
HITS: Binarizing physiological time series with deep hashing neural network
Fu, Z., Wang, C., Wei, G., Zhang, W., Du, S., & Hong, S. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.03.003 · Cited by 5
NetPrune: A sparklines visualization for network pruning
Pommé, L.-E., Bourqui, R., Giot, R., Vallet, J., & Auber, D. (2023) · Visual Informatics · doi:10.1016/j.visinf.2023.04.001 · Cited by 7
Scheduling Machine Learning Compressible Inference Tasks with Limited Energy Budget
Barros, T. D. S., Ferré, D., Giroire, F., Aparicio-Pardo, R., & Pérennès, S. (2024) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3673038.3673106 · Cited by 5
“For an extensive review, see [4].”
Deep Neural Network Model Compression and Signal Processing
Ukil, A., Majumdar, A., Jara, A. J., & da Gama, J. M. P. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10627180 · Cited by 4
“Broadly speaking deep neural network (DNN) model compression can be categorized into four approaches [8], [9], [10], [11], [12], [13]: 1.”
BackgroundYe, D., Zhu, T., Gao, K., Zhu, C., & Zhou, W. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3532351 · Cited by 4
“Model compression techniques can be broadly categorized into four types: parameter pruning and sharing, low-rankfactorization,transferred/compactconvolutionalfilters,and knowledge distillation [27], [28].”
A geometric modeling of Occam’s razor in deep learning
Sun, K., & Nielsen, F. (2025) · Information Geometry · doi:10.1007/s41884-025-00167-2 · Cited by 28
“(19) is measured, and therefore the total volume is measured larger. log V is an O ( D ) terms, meaning the volume grows with the number of dimensions.”
Inference Routing Over Multi-Hop Edge Networks
Xu, C., Liu, Y., & Yang, J. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3613490 · Cited by 5
“Compared to model compression [27] and pruning [28], early-exit DNN does not lead to a decrease of inference accuracy.”
Deep Compression with Low Rank and Sparse Integrated Decomposition
Huang, J., Sun, W., Huang, L., & Chen, S. (2019) · International Conference on Computer Science and Network Technology (ICCSNT) · doi:10.1109/iccsnt47585.2019.8962461 · Cited by 5
“These methods can be broadly classified into four categories [7], namely, parameter pruning and sharing [8]-[11], low-rank factorization [12]-[16], transferred/compact convolutional filters [17], [18] and knowledge distillation [19], [20].”
MethodThepade, S. D., & Ramnani, G. (2021) · International Journal of Healthcare Information Systems and Informatics · doi:10.4018/ijhisi.20211001.oa24 · Cited by 4
“Neural network training is also computationally expensive, requiring a lot of time (Chen et al., 2018), with rework often needed regarding the values of weights, network design, and parameter tuning (Cheng et al., 2018).”
BackgroundCompressing Deep Neural Network: A Black-Box System Identification Approach
Sahu, I., Pal, A., Ukil, A., & Majumdar, A. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533962 · Cited by 7
Transmission Line Planning Based on Artificial Intelligence in Smart Cities
Dong, Y., He, T., Du, B., Shou, W., Zhang, Z., & Zhang, S. (2022) · Mobile Information Systems · doi:10.1155/2022/2010189 · Cited by 4
“[11] applied simulated annealing algorithm, genetic algorithm, and particle swarm optimization algorithm, respectively, to the spatial optimization problem of multiobjective land allocation.”
MethodChoosing Appropriate AI-enabled Edge Devices, Not the Costly Ones
Zhang, Z., Li, F., Lin, C., Wen, S., Liu, X., & Liu, J. (2021) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads53394.2021.00031 · Cited by 7
Shahriari, M. A., Pardo, D., Kargaran, S., & Teijeiro, T. (2023) · Geophysical Journal International · doi:10.1093/gji/ggad249 · Cited by 4
“This is tightly related to model compression [32], a research field that explores methods for reducing the complexity of a reference model without affecting its accuracy.”
Lightweight CNN for HRRP Recognition Based on Attention Mechanism Structured Pruning
Wang, Y., Zhang, Z., Yuan, M., Jiandong, L., & Zhang, L. (2023) · IEEE International Radar Conference · doi:10.1109/radar54928.2023.10371061 · Cited by 4
“The mainstream model compression techniques include network pruning, low-rank decomposition, network quantization and knowledge distillation [5].”
BackgroundBarros, T. D. S., Giroire, F., Aparicio-Pardo, R., Pérennès, S., & Natale, E. (2024) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid59990.2024.00045 · Cited by 9
“A large review [8] investigated model compression techniques (e.g. pruning and knowledge distillation) for Deep Learning (DL).”
BackgroundA New Clustering-Based Technique for the Acceleration of Deep Convolutional Networks
Pikoulis, E. V., Mavrokefalidis, C., & Lalos, A. S. (2020) · International Conference on Machine Learning and Applications · doi:10.1109/icmla51294.2020.00222 · Cited by 11
Accelerating 3D scene analysis for autonomous driving on embedded AI computing platforms
Nousias, S., Pikoulis, E., Mavrokefalidis, C., Lalos, A. S., & Μουστάκας, Κ. (2021) · IFIP/IEEE International Conference on Very Large Scale Integration · doi:10.1109/vlsi-soc53125.2021.9606990 · Cited by 5
“The MCA literature is thoroughly described in several recent survey papers like [13], [14] and [15], where the presented techniques can be utilized on the algorithmic or the hardware level, while, currently, there are also methods that take into account both algorithmic and hardware aspects.”
Continual Learning in Predictive Autoscaling
Hao, H., Chu, Z., Zhu, S., Jiang, G., Wang, Y., Jiang, C., Zhang, J. Y., Jiang, W., Xue, S., & Zhou, J. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615463 · Cited by 11
Self-Paced Knowledge Distillation for Real-Time Image Guided Depth Completion
Wang, S., Hu, M. H., Li, B., & Gong, X. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3154243 · Cited by 5
Successive Pruning for Model Compression via Rate Distortion Theory.Preprint
Isik, B., No, A., & Weissman, T. (2021) · arXiv (Cornell University) · Cited by 6
Dynamical Isometry: The Missing Ingredient for Neural Network PruningPreprint
Wang, H., Qin, C., Bai, Y., & Fu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05916 · Cited by 5
“For more comprehensive literature, we refer interested readers to several surveys: an outdated one [39], some recent surveys of pruning alone [12, 1] or pruning as a sub-topic under the general umbrella of model compression and acceleration [46, 3, 4, 7].”
BackgroundMeneghetti, D. D. R., Homem, T. P. D., de Oliveira, J. H. R., Silva, I. J., Perico, D. H., & Bianchi, R. A. C. (2021) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-021-01336-y · Cited by 18
Preview-Based Category Contrastive Learning for Knowledge Distillation
Ding, M., Wu, J., Dong, X., Li, X., Qin, P., Gan, T., & Nie, L. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3541067 · Cited by 5
“Thus, recent years have witnessed a growing trend of model compression [14]–[18], which aims to simplify the network architecture and decrease the network parameters.”
BackgroundParallel Feature Fusion for Multimodal Scene Recognition on Dual-Core MCUs
Woodward, K., & Kanjo, E. (2025) · IEEE Pervasive Computing · doi:10.1109/mprv.2025.3555334 · Cited by 5
Transfer Learning-Based Super-Resolution for High-Precision Medical Imaging
Lee, J. Y., Hussain, M. I., Lee, K.-H., Shim, H.-S., Han, S., & Yang, D. (2025) · IEEE Access · doi:10.1109/access.2025.3587263 · Cited by 8
Mahalingam, S., Manap, A., Chowdhury, M. S., Afandi, N., Arith, F., & Nugroho, A. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.113115 · Cited by 4
A Structurally Optimized and Efficient Lightweight Object Detection Model for Autonomous Driving
Li, M., Wang, J., Chen, S., Liu, L., Li, K., Zhao, Z., & Yun, H. (2025) · Sensors · doi:10.3390/s26010054 · Cited by 3
“Current network compression strategies can be broadly categorized into four groups: structured pruning, knowledge distillation, post-training quantization, and hand-crafted or NAS-based lightweight backbones [10].”
Khan, T. M., Haq, Q. E. U., Iqbal, S., & Soomro, T. A. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114526 · Cited by 8
Logic Shrinkage: Learned Connectivity Sparsification for LUT-Based Neural Networks
Wang, E., Auffret, M., Stavrou, G.-I., Cheung, P. Y. K., Constantinides, G. A., Abdelfattah, M. S., & Davis, J. J. (2023) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3583075 · Cited by 4
“In order to exploit this in the quest for ever-greater performance within given area and power budgets, researchers and industrial practitioners alike are increasingly turning to low-precision data types [3, 26, 32].”
BackgroundAccordion: A Communication-Aware Machine Learning Framework for Next Generation Networks
Ayed, F., Domenico, A. D., García‐Rodríguez, A., & López‐Pérez, D. (2023) · IEEE Communications Magazine · doi:10.1109/mcom.001.2200358 · Cited by 7
“The main techniques used to reduce the communication requirements and the computational complexity of AI/ML models are [10], [11]:”
MethodVeriPrune: Equivalence verification of node pruned neural network
Wang, W., Wang, K., Cheng, Z., & Yang, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127347 · Cited by 4
EEG Emotion Recognition Based on Self-Distillation Convolutional Graph Attention Network
Hao, C., & Feng, S. (2024) · ICST Transactions on e-Education and e-Learning · doi:10.4108/eetel.4974 · Cited by 3
“The idea of using larger models to guide the training of smaller models was first proposed by Bucilua et al. for the compression of ensemble models [25].”
MethodDTS: dynamic training slimming with feature sparsity for efficient convolutional neural network
Yin, J., Wang, W., Guo, Z., & Ji, Y. (2024) · Journal of Real-Time Image Processing · doi:10.1007/s11554-024-01511-y · Cited by 3
“In addition, most scientific studies [4–6] have shown that deep CNNs have significant parameter redundancy.”
Special Session: Estimation and Optimization of DNNs for Embedded Platforms
Jantsch, A., Han, S., Meng, L., Bringmann, O., Tang, H., Yang, S., Li, H., Wess, M., & Lechner, M. (2024) · International Conference on Hardware/Software Codesign and System Synthesis · doi:10.1109/codes-isss60120.2024.00013 · Cited by 3
Star-YOLOv8s: an ultra-lightweight model for optimal blueberry picking
Wang, J., Lin, H., Hu, B., Xu, C., Zhang, B., Wang, H., & Yang, F. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/ade271 · Cited by 3
“Moreover, smaller models are generally more efficient in terms of both energy consumption and inference time, making them better suited for real-time applications, such as detecting blueberry ripeness in agricultural automation systems [56].”
Lightweight Quadruple-GAN for Interference Source Recognition in Wi-Fi Networks
Jiang, T., Qin, X., Xu, X., Chen, J., & Dong, W. (2020) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc51575.2020.9345180 · Cited by 3
“Many compression techniques have emerged for neural network compression [8], such as parameter pruning and sharing [9], low-rank factorization [10], compact convolutional filters [11]-[12] and knowledge distillation (KD) [13].”
BackgroundEvaluation of Compressed Residential Energy Forecasting Models
Fagaras, R., Nichiforov, C., Stamatescu, I., & Stâmâtescu, G. (2021) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc52423.2021.9658990 · Cited by 5
“Four categories of techniques for the compression of neural networks are identified in [11]: weight sharing and pruning, low rank factorization of parameter tensors, design of compact convolutional filters, knowledge distillation.”
MethodDeep Unfolded Sparse Refinement Network Based Detection in Uplink Massive MIMO
Datta, A., Nema, A., & Bhatia, V. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3166399 · Cited by 6
“However, as the above DL based detectors do not perform any model compression techniques [14], [15], they utilize fully connected DL networks resulting in significant computation resources and energy costs.”
BackgroundGradFreeBits: Gradient-Free Bit Allocation for Mixed-Precision Neural Networks
Bodner, B. J., Shalom, G. B., & Treister, E. (2022) · Sensors · doi:10.3390/s22249772 · Cited by 5
“However, as with other compression methods, this typically comes at the cost of a reduced performance [5].”
BackgroundMethodLin, Y., Tang, J., Gao, B., Zhang, Q., Qian, H., & Wu, H. (2022) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2022.3231471 · Cited by 4
“The sensitivity of network performance to each weight is not the same [11].”
BackgroundMembership inference attacks against compression models
Jin, Y., Lou, W., & Gao, Y. (2023) · Computing · doi:10.1007/s00607-023-01180-y · Cited by 4
EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Models
Li, X., Parazeres, M., Oberman, A. M., Ghaffari, A., Asgharian, M., & Nia, V. P. (2023) · SN Computer Science · doi:10.1007/s42979-023-01921-y · Cited by 6
“Compressing deep learning models comes at the costs of accuracy loss, and increasing training time (to a greater extent on quantized networks) [9, 10].”
Residual encoding framework to compress DNN parameters for fast transfer
Zhou, L., Song, R., Chen, G., Festag, A., & Knoll, A. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110815 · Cited by 4
An Approach to the Systematic Characterization of Multitask Accelerated CNN Inference in Edge MPSoCs
Cilardo, A., Maisto, V., Mazzocca, N., & di Torrepadula, F. R. (2023) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3611015 · Cited by 6
“Cheng et al. [3] further categorized model compression techniques into the following four groups: • Parameter pruning and sharing attempts to reduce the number of parameters by removing the redundant and noncritical ones [10].”
BackgroundTowards Efficient Edge Learning for Large Models in Heterogeneous Resource-limited Environments
Liu, D., Wang, Z., Pang, X., Sun, Y., Hu, J., Sun, P., & Hu, Y. (2023) · International Conference on Big Data Computing and Communications (BigCom) · doi:10.1109/bigcom61073.2023.00038 · Cited by 5
Lightweight Model for Occlusion Removal from Face Images
John, S., & Danti, A. (2024) · Annals of Emerging Technologies in Computing · doi:10.33166/aetic.2024.02.001 · Cited by 3
Trainable pruned ternary quantization for medical signal classification models
Vindas, Y., Guépié, B. K., Almar, M., Roux, E., & Delachartre, P. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128216 · Cited by 5
Chen, L., Shi, X., Zhou, L., Wang, Y., Ma, C., & Zhu, W. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5498-4_3 · Cited by 3
Nonlinear equalization based on feature crosses neural networks for High-speed PAM4 transmission
Yang, R., Jiang, J., Zhang, Q., Xin, X., Yao, H., Gao, R., Tian, F., Tian, Q., Wang, F., Li, Z., et al. (2024) · Optics Communications · doi:10.1016/j.optcom.2024.130976 · Cited by 3
Fan Blade Crack Detection Algorithm Based on Multi-Scale Feature Fusion
Qi, Y., Tang, H., & Altangerel, K. (2025) · IEEE Access · doi:10.1109/access.2025.3532077 · Cited by 3
“Once a fault occurs, the loss will be large [16].”
Survey of Lightweight Object Detection Model Methods
Yang, C., Gao, Z., Hao, Y., & Yang, R. (2024) · International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC) · doi:10.1109/icairc64177.2024.10899985 · Cited by 3
Emerald, S. C., & Vengattaraman, T. (2025) · International Journal of Information Technology · doi:10.1007/s41870-025-02545-6 · Cited by 3
“The proliferation of deep learning (DL) techniques in diverse applications, namely computer vision, natural language processing (NLP), and signal processing [1], has led to an increasing demand for efficient neural network (NN) architectures capable of operating under resource constraints [2].”
Universal and Succinct Source Coding of Deep Neural Networks
Basu, S., & Varshney, L. R. (2022) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2023.3261819 · Cited by 3
“To ameliorate this storage bottleneck, we have developed lossless compression algorithms for feedforward deep neural networks that make use of their particular structural invariances in inference and can act as a final stage for other lossy techniques [21].”
MethodMax-Variance Convolutional Neural Network Model Compression
Boone-Sifuentes, T., Robles‐Kelly, A., & Nazari, A. (2020) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta51227.2020.9363347 · Cited by 3
“As mentioned in [5], existing methods can be broadly classified into parameter pruning [6], low rank factorization [7], knowledge distillation [8] and compaction of convolutional filters [9].”
MethodZheng, X., Zhang, L., Hui, B., Tian, L., & Cai, Z. (2022) · IEEE Internet of Things Magazine · doi:10.1109/iotm.001.2100194 · Cited by 3
High-precision regressors for particle physics
Bishara, F., Paul, A., & Dy, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-52941-4 · Cited by 5
Ananthabhotla, I., Ewert, S., & Paradiso, J. A. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207053 · Cited by 3
“In this context, model weight quantization [2], memory-efficient architectures [3], [4], parameter pruning and sharing strategies [5], and student-teacher training [6] were successfully employed to reduce resources while trying to maintain modeling power.”
Method
Metadata
- Authors
- Cheng, Y., Wang, D., Zhou, P., & Zhang, T.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- Harbin Institute of Technology (CN)
- Huazhong University of Science and Technology (CN)
- Tsinghua University (CN)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2018-01
- Volume
- 35
- Issue
- 1
- Pages
- 126-136
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W2783538964
- Cited by
- 817
- References
- 67
- Citations by year
- 2026: 114 · 2025: 132 · 2024: 116 · 2023: 110 · 2022: 109 · 2021: 92 · 2020: 89 · 2019: 41 · 2018: 14
- FWCI
- 29.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- model compression, deep neural network, inference acceleration, parameter reduction, convolutional neural networks, training efficiency, GPU computing, face verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.