Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges
The DOI matches its Crossref record; 61 of the 67 references it cites were found in a registry or an index.
Abstract
In recent years, deep neural networks (DNNs) have received increased attention, have been applied to different applications, and achieved dramatic accuracy improvements in many tasks. These works rely on deep networks with millions or even billions of parameters, and the availability of graphics processing units (GPUs) with very high computation capability plays a key role in their success. For example, Krizhevsky et al. achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully connected layers. Usually, it takes two to three days to train the whole model on the ImagetNet data set with an NVIDIA K40 machine. In another example, the top face-verification results from the Labeled Faces in the Wild (LFW) data set were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally connected, and fully connected layers. It is also very time-consuming to train such a model to obtain a reasonable performance. In architectures that only rely on fully connected layers, the number of parameters can grow to billions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [26]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Recently, more state-of-the-art architectures are used as baseline models in many works, including network in networks [67], VGGNets [68], and ResNets [69].”
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“[1] achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully-connected layers.”
MethodInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The work in [51] introduced a KD compression framework, which eased the training of deep networks by following a student-teacher paradigm, in which the student was penalized according to a softened version of the teacher’s output.”
BackgroundDeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · cited 1×
“example is the top face verification results on the Labeled Faces in the Wild (LFW) dataset were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally-connected, and fully-connected layers [2], [3].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“works that directly train CNNs with binary weights, for instance, BinaryConnect [13], BinaryNet [14] and XNORNetworks [15].”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
Method
Cited by
Items 501–600 of 818
Efficient architecture for deep neural networks with heterogeneous sensitivity
Cho, H., Jang, J., Lee, C., & Yang, S. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.10.017 · Cited by 3
Joint Structure and Parameter Optimization of Multiobjective Sparse Neural Network
Huang, J., Sun, W., & Huang, L. (2021) · Neural Computation · doi:10.1162/neco_a_01368 · Cited by 4
Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater Objects
Er, M. J., Chen, J., & Zhang, Y. (2023) · Artificial intelligence · doi:10.5772/intechopen.107409 · Cited by 4
ExplainFix : Explainable spatially fixed deep networks
Gaudio, A., Faloutsos, C., Smailagic, A., Costa, P., & Campilho, A. (2022) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1483 · Cited by 3
“Neural networks are highly compressible (Cheng et al., 2018).”
Active Learning for Multiple Target Models
Huang, S.-J., Li, Y., & Tang, Y.-P. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3602682 · Cited by 6
Modularized Multi-Stage Binarized Deep-Model Compression and Performance Recovery
Lu, C.-H., Wang, J.-W., & Teng, C.-H. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3002340 · Cited by 3
“However, use of this compression method often compromises accuracy [23].”
MethodPixelHop++: A Small Successive-Subspace-Learning-Based (SSL-based) Model for Image ClassificationPreprint
Chen, Y., Rouhsedaghat, M., You, S. O., Rao, R., & Kuo, C. ‐. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03141 · Cited by 3
“topic in recent years since small models are essential to mobile and edge computing applications. There has been a lot of research dedicated to neural network model compression and acceleration, e.g. [1, 2, 3, 4, 5]. Techniques such as parameter pruning, quantization, binarization and sharing, low-rank factorization,…”
MethodEr, M. J., Chen, J., Zhang, Y., & Gao, W. (2022) · doi:10.36227/techrxiv.19350389.v1 · Cited by 3
Benarfa, A., Dahmane, S., Brik, B., & Yagoubi, M. B. (2024) · Internet Technology Letters · doi:10.1002/itl2.630 · Cited by 2
Enhancing knowledge distillation via genetic recombination
Cao, Y., Zhou, C., Liu, M., Luo, W., & Luo, X. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113414 · Cited by 2
Reducing Carbon Footprint of Machine Learning Through Model Compression and Pruning
Stow, M. T., & Stewart, A. A. (2025) · doi:10.38124/ijisrt/25aug970 · Cited by 5
Ye, X., Wang, H., & Yao, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112719 · Cited by 4
Benchmarking deep learning models for ET0 forecasting on edge devices
Luo, K., Lim, C. S., Ishak, M. H. I., Mahmud, M. S. A., & Ba, N. (2025) · Hydrology research · doi:10.2166/nh.2025.130 · Cited by 5
Building Applied Models From Multiple Sources
Subrahmanyam, S. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-3301-4.ch002 · Cited by 2
NJ-Unet: A novel Unet-based greenhouse tomato leaf segmentation neural network and its application
Zhang, K., Gao, R., Shen, A., Wang, Y., Pang, G., Yang, L., Liu, X., Liu, X., & Sun, J. (2026) · Information Processing in Agriculture · doi:10.1016/j.inpa.2026.01.007 · Cited by 2
FPGA-Enabled Deep Learning for Real-Time DC Series Arc Fault Detection
Li, Y., Mao, Y., Rothmann, M., Weiss, R. J., & Porrmann, M. (2026) · IEEE Open Journal of Industry Applications · doi:10.1109/ojia.2026.3671845 · Cited by 3
Design of Noise Shaping Quantizers for Data Compaction of Neural Networks
Minami, Y., Ikeda, T., & Ishikawa, M. (2020) · Transactions of the Society of Instrument and Control Engineers · doi:10.9746/sicetr.56.425 · Cited by 2
Domain Adaptation Regularization for Spectral Pruning
Dillard, L., Shinya, Y., & Suzuki, T. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.119 · Cited by 2
Cloud Detection for Pleiades and SPOT 6/7 Imageries Using Modified K-means and Deep Learning
Prabowo, Y., Candra, D. S., & Maulana, R. (2021) · International Journal on Advanced Science Engineering and Information Technology · doi:10.18517/ijaseit.11.6.14312 · Cited by 2
“The output of convolution will be passed through the activation function, making the output non-linear [34].”
BackgroundHKDP: A Hybrid Approach On Knowledge Distillation and Pruning for Neural Network Compression
Hongle, C., Qirui, S., Chen, J., & Wen, Q. (2021) · International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · doi:10.1109/iccwamtip53232.2021.9674054 · Cited by 2
“As a result, many methods of compressing large models have been developed [12].”
MethodReal-Time and Non-Contact Arrhythmia Recognition Algorithm for Hardware Implementation
Lei, K.-Z., Ku, M.-Y., & Lee, S.-Y. (2022) · IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE) · doi:10.1109/rasse54974.2022.9989597 · Cited by 2
“Our proposed system uses a network model compression method based on AI knowledge distillation [10].”
MethodVision-Based Real-Time Obstacle Detection System for Trains
Han, Z., Zhou, M., Lu, X., Xue, D., & Feng, R. (2022) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2022-01-7092 · Cited by 2
Distracted driving behavior recognition based on improved MobileNetV2
Bai, X., Li, J., Zhang, C., Hu, H., & Gu, D. (2023) · Journal of Electronic Imaging · doi:10.1117/1.jei.32.5.053021 · Cited by 3
Wen, X., Liang, B., Zhao, B., Hu, X., Yuan, M., Hu, W., Liu, T., Yang, Y., & Xing, D. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1204044 · Cited by 2
TRAIN: A Reinforcement Learning Based Timing-Aware Neural Inference on Intermittent Systems
Cheng, S.-T., Lim, W. S., Tu, C.-H., & Chang, Y.-H. (2023) · doi:10.1109/iccad57390.2023.10323634 · Cited by 4
Cao, X., Yang, R., Guo, C., & Qin, H. (2024) · Sensors · doi:10.3390/s24010272 · Cited by 2
“For most mechanical devices, such as motor bearings, self-priming centrifugal pumps, and axial piston hydraulic pumps [22–26], FD typically entails the analysis of sensor data collected during actual operations.”
BackgroundDadà, M., Zulberti, L., Nannipieri, P., Fanucci, L., & Moranti, S. (2023) · European Data Handling & Data Processing Conference (EDHPC) · doi:10.23919/edhpc59100.2023.10396304 · Cited by 4
Fast On-Device Learning Framework for Single-Image Super-Resolution
Lee, S. H., Park, K., Cho, S., Lee, H.-S., Choi, K., & Cho, N. I. (2024) · IEEE Access · doi:10.1109/access.2024.3375120 · Cited by 4
“Quantizing deep learning models is essential for their use in resource-constrained environments that require integer-arithmetic computations [54], [55].”
BackgroundA novel iteration scheme with conjugate gradient for faster pruning on transformer models
Li, J., Zhu, Y., & Sun, K. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01595-w · Cited by 2
“In response to the challenges, pruning has emerged as a common and efficient technique for reducing model sizes and complexity [19–21].”
Pruning Policy for Image Classification Problems Based on Deep Learning
Pachón, C. G., Pinzon-Arenas, J. O., & Ballesteros, D. M. (2024) · Informatics · doi:10.3390/informatics11030067 · Cited by 2
Performance prediction of IPMC modified with SiO2-SGO based on backpropagation neural network
Zhai, Z., Tian, A., Zhang, X., Du, H., & Wang, Y. (2024) · Nanotechnology and Precision Engineering · doi:10.1063/10.0028830 · Cited by 2
Putro, N. A. S., Leu, J., Ananto, N., Avian, C., Mahali, M. I., & Prakosa, S. W. (2024) · IEEE Embedded Systems Letters · doi:10.1109/les.2024.3487236 · Cited by 2
“While KD is widely explored for classi-fication problems, its application is often limited or can not be implemented in some regression tasks [10].”
Lee, K., Cha, M., Kim, S., Nguyen, X. T., & Lee, H. (2025) · International Conference on Consumer Electronics · doi:10.1109/icce63647.2025.10930191 · Cited by 8
“However, the large computational complexity of CNN models incurs extensive computation time and power, while the high space complexity for storing model weights requires frequent DDR access [2].”
Musa, A., Hamada, M., Hassan, M., Umar, U., Aaron, A., & Mahendran, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-81083-1_1 · Cited by 2
Camera clustering for scalable stream-based active distillation
Manjah, D., Cacciarelli, D., Vleeschouwer, C. D., & Macq, B. M. M. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128408 · Cited by 2
NUPES: Non-Uniform Post-Training Quantization via Power Exponent Search
Yvinec, E., Dapogny, A., & Bailly, K. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3593987 · Cited by 13
Online Learning from data streams via decentralized and asynchronous SGD
Tosi, M. D. L., & Theobald, M. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.108052 · Cited by 3
“Consequently, training a large model with SGD may be a time-consuming task [10].”
Segment Anything Model for Gastric Cancer
Li, L., Wang, C., Geng, Y., Chen, T., Wang, Z., Lin, K., Wei, H., Wang, J., Wang, D., Niu, D., et al. (2025) · Cancer Medicine · doi:10.1002/cam4.71246 · Cited by 2
“Knowledge distillation [26–28] is widely recognized as an effective model compression technique, enabling the adaptation of complex models to hardware with limited processing capabilities.”
Self-citationGao, C., Chen, X., Wang, J., Wang, J., & Yang, G. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111914 · Cited by 3
Traffic Flow Correlation Analysis of K Intersections Based on Deep Learning
Chu, H., Wang, C.-K., & Liao, Y.-X. (2018) · Smart innovation, systems and technologies · doi:10.1007/978-3-030-03748-2_24 · Cited by 2
Bit-Quantized-Net: An Effective Method for Compressing Deep Neural Networks
Li, C., Du, Q., Xu, X., Zhu, J., & Chu, D. (2021) · Mobile Networks and Applications · doi:10.1007/s11036-020-01687-0 · Cited by 2
Ghosh, R. (2020) · IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc51696.2020.9668874 · Cited by 2
Schuster, A., Heidorn, C. J. A., Brand, M., Keszöcze, O., & Teich, J. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-93736-2_29 · Cited by 3
Lightweight jet reconstruction and identification as an object detection task
Pol, A. A., Aarrestad, T. K., Govorkova, E., Halily, R., Klempner, A., Kopetz, T., Lončar, V., Ngadiuba, J., Pierini, M., Sirkin, O., et al. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac7a02 · Cited by 4
Knowledge Distillation in Granular Fuzzy Models by Solving Fuzzy Relation Equations
Rakytyanska, H. B. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-32095-8_4 · Cited by 5
Mobiprox: Supporting Dynamic Approximate Computing on Mobiles
Fabjančič, M., Machidon, O. M., Sharif, H., Zhao, Y., Misailovíc, S., & Pejović, V. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3365957 · Cited by 4
“Efforts have, thus, focused on reducing the complexity, while preserving the inference power of DL models through weight quantization [6], pruning [7], [19], [20], knowledge distillation [8] and other methods [21].”
BackgroundSemi-supervised and Population Based Training for Voice Commands Recognition
Elibol, O. H., Keskin, G., & Thomas, A. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683265 · Cited by 2
“Usually this is done through compression techniques [9].”
MethodTraining deep neural networks for wireless sensor networks using loosely and weakly labeled images
Zhou, Q., Chen, Y., Li, B., Li, X., Zhou, C., Huang, J., & Hu, H. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.09.040 · Cited by 3
“ecause fewer parameters mean higher utilization of key parameters and usually lead to less computational costs, we investigated model compression approaches that can be separated into four categories [30]: 1) Parameter Pruning and Sharing [31], 2) Low-Rank Factorization [32], 3) Transferred Convolutional Filters [33],…”
A novel weight initialization with adaptive hyper-parameters for deep semantic segmentation
Haq, N. U., Khan, A., Rehman, Z. U., Din, A., Shao, L., & Shah, S. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10510-1 · Cited by 2
A data-aware dictionary-learning based technique for the acceleration of deep convolutional networks
Pikoulis, E., Mavrokefalidis, C., & Lalos, A. S. (2021) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp53017.2021.9733527 · Cited by 2
“The MCA-related literature has been increasing in recent years and there are numerous surveys that provide a comprehensive overview of the area ([2], [9], [10], [11]).”
BackgroundDeep Learning Development Status in China
Center for Electronics and Information Studies, C. A. O. E. (2020) · Springer eBooks · doi:10.1007/978-981-15-4584-9_2 · Cited by 2
Global Capacity Measures for Deep ReLU Networks via Path SamplingPreprint
Theisen, R., Klusowski, J. M., Wang, H., Keskar, N. S., Xiong, C., & Socher, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10245 · Cited by 2
Optimizing Memory Placement using Evolutionary Graph Reinforcement LearningPreprint
Khadka, S., Aflalo, E., Marder, M., Ben-David, A., Miret, S., Mannor, S., Hazan, T., Tang, H., & Majumdar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07298 · Cited by 7
“n our policies end-to-end on the NNP-I chipset to ensure that the solutions are robust to the real-world constraints and uncertainties of the chip. Complimentary to other approaches like compression [Cheng et al., 2018, Kim et al., 2015], sparsification [Venkatesh et al., 2017] and network pruning [Han et al., 2015], E…”
MethodVariance-Guided Structured Sparsity in Deep Neural Networks
Pandit, M. K., & Banday, M. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3221688 · Cited by 1
Compressing Neural Networks with Two-Layer Decoupling
Jonghe, J. D., Usevich, K., Dreesen, P., & Ishteva, M. K. (2023) · IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing · doi:10.1109/camsap58249.2023.10403509 · Cited by 4
“Consequently, significant research efforts have focused on compressing and accelerating neural networks such as low-rank factorization methods, filter and weight pruning and knowledge distillation [2], [5], [9], [14].”
BackgroundKnowledge Distillation Scheme for Named Entity Recognition Model Based on BERT
Ye, C., & Hernandez, A. A. (2023) · International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI) · doi:10.1109/mlbdbi60823.2023.10482264 · Cited by 2
“In order to optimize deep neural network models and reduce their computational complexity and storage requirements, recent work has mainly focused on two directions: designing more efficient deep model architectures [1-2] and developing model compression and acceleration techniques [3].”
MethodD2D-assisted two-stage model aggregation scheme based on over-the-air computation
ZHANG, R., Deng, N., Wei, H., Xing, C., & Nan, Z. H. A. O. (2024) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0402 · Cited by 1
Approximate Convolutional Neural Networks Based on Dynamic Binding Windows
Adelkhani, M., & Mehdi, S. (2024) · CPSSI International Symposium on Cyber-Physical Systems (Applications and Theory) (CPSAT) · doi:10.1109/cpsat64082.2024.10745458 · Cited by 1
Light-Weighted Temporal Evolution Inference for Generative Face Video Compression
Zhang, Z., Chen, B., Yin, S., Wang, S., & Ye, Y. (2024) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp61759.2024.10743340 · Cited by 4
A Low-Cost Computational Spectrometer Based on a Trained Sparse Base Matrix
Gao, Y., Pan, H., Sheng, Y., Rui, W., Zheng, Y., & Yang, L. (2025) · Sensors · doi:10.3390/s25030953 · Cited by 1
“…a reconstruction algorithm based on deep learning can achieve better reconstruction results, it places high demands on the storage and computational capabilities of the platform, which makes the embedded integration impossible on many hardware devices and in situ measurements difficult [20].”
TSP: Task-Specific Pruning for Personalized Image Classification on Edge Devices
Wang, Y., Shi, B., & Zhang, H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890691 · Cited by 1
ChannelZip: SLO-Aware Channel Compression for Task-Adaptive Model Serving on IoT Devices
Luo, P., Hou, J., Tan, H., Yuan, M., Wu, G., Guo, K., Wang, Z., & Li, X. (2025) · ACM Transactions on Sensor Networks · doi:10.1145/3729534 · Cited by 1
“A rich literature has explored many model compression approaches, including the design of lightweight model architecture [3, 11], parameter quantization [2, 22], tensor decomposition [53], and network pruning [4, 24, 36].”
Optimized Fuzzy Stacked Ensemble Model-Based Diabetes Prediction in Pakistan
Riaz, A., Khan, N., & Riaz, S. (2025) · Journal of Agricultural Biological and Environmental Statistics · doi:10.1007/s13253-025-00688-4 · Cited by 1
“Feedforward neural networks, also referred to as multilayer neural networks, are artificial neural networks where nodes do not create loops (Cheng et al. 2018).”
FPGA Based Artificial Neural Network Accelerator
Fatima, E., Fahad, M., Abrar, H., Haroon-Ur-Rashid, & Waris, H. (2024) · International Multi-Topic Conference · doi:10.1109/inmic64792.2024.11004346 · Cited by 1
Global Structural Knowledge Distillation for Semantic Segmentation
Park, H., Ahn, K.-H., Choi, H., & Min, D. (2025) · IEEE Access · doi:10.1109/access.2025.3575066 · Cited by 2
“To address this, various model compression techniques, such as network pruning [6], [7], network quantization [8], [9], and knowledge distillation (KD) [10], [11], [12], have been explored to reduce the model FIGURE 1.”
Learnable Sparse Customization in Heterogeneous Edge Computing
Xue, J., Sun, S., Liu, M., Wang, Y., Liu, Z., & Wang, J. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00012 · Cited by 4
Zhang, C., Yin, W., Zhang, C., Peng, K., & Zhang, X. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113679 · Cited by 3
Deng, C., Cheng, J., Su, Y., An, Z., Yang, Z., Xia, Z., Zhang, Y., & Wang, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108136 · Cited by 4
A Comprehensive Survey on Data Distillation: Techniques, Frameworks, and Future Directions
Razi, Q., Singh, S., Priyadarshini, R., Hassija, V., & Chalapathi, G. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3633710 · Cited by 1
“KD has been widely applied in model compression [30], [31] and ensemble learning [32].”
HEMQ: Towards More Accurate Post-training Mixed-Precision Quantization of Neural Networks
Yuan, Y., Shi, H., & Gao, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4367-0_43 · Cited by 1
Based on Tensor Core Sparse Kernels Accelerating Deep Neural Networks
Lv, S., Liu, D., Yang, L. T., Peng, X. T., Zhao, R., Yang, Z., & Feng, J. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3637268 · Cited by 1
Edge AI Revolution: Compressed Neural Networks Enabling Real-Time Mobile Processing
Kanwar, G. (2025) · European Modern Studies Journal · doi:10.59573/emsj.9(5).2025.11 · Cited by 1
Lee, J., Chung, H., Suh, M., Lee, J.-H., & Choi, K. S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0339064 · Cited by 6
“For resource-constrained settings, established techniques like transfer learning [31,32] and model compression [33,34] could be used in conjunction with our framework to further reduce computational demands.”
Trabes, E., Dellel, K., Valderrama, C. A., & Ahmed, A. T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-10192-1_35 · Cited by 1
Chen, Y., Wang, J., Fang, Z., Ni, T., Geng, J., & Ge, W. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3654662 · Cited by 1
“Research has shown that a significant proportion of parameters within neural networks are redundant, which provides a possibility for lightweight [31], [32].”
Lin, H., Yang, L., Zhao, X., Tang, S., Huang, J., & Qi, F. (2025) · International Conference on Cyber Security and Information Engineering (ICCSIE) · doi:10.1109/iccsie65832.2025.00029 · Cited by 1
“To address computational bottlenecks, researchers have explored model compression techniques such as pruning, quantization, and knowledge distillation, as well as lightweight Large Language Model (LLM) architectures designed for edge scenarios[8].”
Kiflie, M. A. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200638 · Cited by 2
Mehrdad, V., Talebzadeh, R., & Fazaeli, N. (2026) · Scientific Reports · doi:10.1038/s41598-026-39926-1 · Cited by 2
Design, assessment, and application of machine learning potential energy surfaces
Andreichev, V., Aydin, S., Töpfer, K., Meuwly, M., & Vazquez-Salazar, L. I. (2026) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae47b9 · Cited by 6
Zia, M. F., Ouameur, M. A., Bagaa, M., Massicotte, D., & Ksentini, A. (2026) · Computer Networks · doi:10.1016/j.comnet.2026.112142 · Cited by 3
Multimodal Remote Sensing Sensors for Edge-Aware Intelligent Perception Systems
Sukhroop, P., Shekhar, Bhardwaj, V. K., Essa, I. M., AL-Shaeli, I. K., & Gharban, H. A. J. (2025) · International Conference on Advanced Computing and Emerging Technologies (ACET) · doi:10.1109/acet67282.2025.11430375 · Cited by 1
Guo, Y., Yao, J., & Liu, X. (2026) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2026.3676094 · Cited by 1
Theodorakopoulos, L., & Theodoropoulou, A. (2026) · Electronics · doi:10.3390/electronics15071405 · Cited by 1
On the Utility of Foundation Models for Fast MRI : Vision‐Language‐Guided Image Reconstruction
Feng, R., He, X., Mercer, R. W., Stewart, Z. E., & Liu, F. (2026) · Magnetic Resonance in Medicine · doi:10.1002/mrm.70374 · Cited by 4
“Third, the structured representations learned by foundation models could inspire the design of reconstruction architectures that better capture multi-scale and contextual dependencies, potentially enhancing both reconstruction accuracy and interpretability.”
Manzoor, H. U., Shabbir, A., Rais, R. N. B., Hussain, S., & Zoha, A. (2026) · International Journal of Intelligent Systems · doi:10.1155/int/8810907 · Cited by 2
“To address this constraint, model compression techniques [68] are widely employed to reduce computational and memory requirements in FL. Key approaches, including pruning [45], sparsifcation [69], gradient compression [69], and quantization [70], enhance the computational efciency of both training…”
A Lightweight Tomato Maturity Detection Method Based on EMBS-DETR
Yan, H., Bao, G., Du, Y., Wu, Q., Zheng, H., & Liu, J. (2026) · Agronomy · doi:10.3390/agronomy16111048 · Cited by 1
Solving the Capacity Mismatch Problem: Self-Knowledge Distillation for Ultra-Lightweight Networks
Acharya, S., Bolla, B. K., & Jojo, J. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.06.301 · Cited by 1
Zarrouk, Y., Bourhaleb, M., Rahmoun, M., Hamdaoui, H., & Hachami, K. (2026) · Array · doi:10.1016/j.array.2026.101069 · Cited by 1
Zhou, G., Wang, J., Mei, Y., Zhou, X., Wang, C. P., Zhang, Y., Zhou, Z., & Liou, Y. (2026) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2026.3685716 · Cited by 1
Jung, H. W. (2019) · doi:10.5121/csit.2019.91711 · Cited by 1
Robust Hypersphere-based Weight Imprinting for Few-Shot Learning
Passalis, N., Iosifidis, A., Gabbouj, M., & Tefas, A. (2020) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco47968.2020.9287340 · Cited by 1
“Despite the success of DL on the aforementioned areas, DL models require powerful hardware both during the training process, as well as during the inference [5], [6].”
BackgroundDC-MPQ: Distributional Clipping-based Mixed-Precision Quantization for Convolutional Neural Networks
Lee, S., & Kim, H. (2022) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas54282.2022.9869959 · Cited by 1
Accelerating Computer Vision Tasks on GPUs using Ramanujan Graph Product Framework
Shaik, F. A., Konduru, T., Varma, G., & Kothapalli, K. (2023) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3570991.3571044 · Cited by 1
TinyML Techniques for running Machine Learning models on Edge Devices
Mukherjee, A., Ukil, A., Dey, S., & Kulkarni, G. (2022) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1145/3564121.3564812 · Cited by 3
“Hence, model compression is of utmost importance to put such models to use with minimum penalty in tiny edge devices [3, 4, 6, 10, 14, 17, 21].”
BackgroundCustomisable Processing of Neural Networks for FPGAs
Denholm, S., & Luk, W. W. (2023) · International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies (HEART) · doi:10.1145/3597031.3597041 · Cited by 1
“…neural networks on FPGAs, optimisations are typically achieved by: (1) reducing data widths and routing through methods such as quantisation [6] [7], sparsely connected networks [13], and weight pruning [3]; and (2) sharing resources amongst the independent neurons via simple queuing and…”
MethodZhu, L. (2023) · Applied Mathematics and Nonlinear Sciences · doi:10.2478/amns.2023.2.00530 · Cited by 1
Efficient Incremental Text-to-Speech on GPUs
Du, M., Liu, C., Qi, J., & Lai, J. (2023) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc58517.2023.10317423 · Cited by 2
“To the best of our knowledge, many incremental TTS pipelines deployed in production are CPU-based, which typically requires reducing the computational cost by compressing[26], sparsifying[27], and distilling[28] the model to obtain real-time synthesis.”
MethodSong, M., Xiong, Z., Zhang, Z., Ren, J., Li, M., Xiao, S., & Tang, Y. (2024) · Journal of Vibroengineering · doi:10.21595/jve.2023.23412 · Cited by 1
“This technology can be divided into parameter pruning and quantization, low-rank decomposition, transfer/compact convolution filter and knowledge distillation [16].”
MethodPruning Based on Activation Function Outputs for Efficient Neural Networks
KAMMA, K., & Wada, T. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10393934 · Cited by 3
“One such technique is structured pruning that removes unimportant neurons (or channels) to make the weight tensor smaller [3].”
MethodFast Super-Resolution Network via Dynamic Path Selection
Jia, L., Hu, Y., Tian, X., & Luo, W. (2023) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml60161.2023.10424923 · Cited by 1
Metadata
- Authors
- Cheng, Y., Wang, D., Zhou, P., & Zhang, T.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- Harbin Institute of Technology (CN)
- Huazhong University of Science and Technology (CN)
- Tsinghua University (CN)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2018-01
- Volume
- 35
- Issue
- 1
- Pages
- 126-136
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W2783538964
- Cited by
- 817
- References
- 67
- Citations by year
- 2026: 114 · 2025: 132 · 2024: 116 · 2023: 110 · 2022: 109 · 2021: 92 · 2020: 89 · 2019: 41 · 2018: 14
- FWCI
- 29.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- model compression, deep neural network, inference acceleration, parameter reduction, convolutional neural networks, training efficiency, GPU computing, face verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.