Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges
The DOI matches its Crossref record; 61 of the 67 references it cites were found in a registry or an index.
Abstract
In recent years, deep neural networks (DNNs) have received increased attention, have been applied to different applications, and achieved dramatic accuracy improvements in many tasks. These works rely on deep networks with millions or even billions of parameters, and the availability of graphics processing units (GPUs) with very high computation capability plays a key role in their success. For example, Krizhevsky et al. achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully connected layers. Usually, it takes two to three days to train the whole model on the ImagetNet data set with an NVIDIA K40 machine. In another example, the top face-verification results from the Labeled Faces in the Wild (LFW) data set were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally connected, and fully connected layers. It is also very time-consuming to train such a model to obtain a reasonable performance. In architectures that only rely on fully connected layers, the number of parameters can grow to billions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [26]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Recently, more state-of-the-art architectures are used as baseline models in many works, including network in networks [67], VGGNets [68], and ResNets [69].”
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“[1] achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully-connected layers.”
MethodInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The work in [51] introduced a KD compression framework, which eased the training of deep networks by following a student-teacher paradigm, in which the student was penalized according to a softened version of the teacher’s output.”
BackgroundDeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · cited 1×
“example is the top face verification results on the Labeled Faces in the Wild (LFW) dataset were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally-connected, and fully-connected layers [2], [3].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“works that directly train CNNs with binary weights, for instance, BinaryConnect [13], BinaryNet [14] and XNORNetworks [15].”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
Method
Cited by
Items 201–300 of 818
Variational Student: Learning Compact and Sparser Networks In Knowledge Distillation Framework
Hegde, S., Prasad, R., Hebbalaguppe, R., & Kumar, V. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054157 · Cited by 23
Real-Time Aeromagnetic Compensation With Compressed and Accelerated Neural Networks
Jiao, J., Yu, P., Zhao, X. H., & Bi, F. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2022.3142007 · Cited by 26
Sepahvand, M., & Abdali-Mohammadi, F. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105413 · Cited by 23
Varga, Z., Shu, Y., Ning, J., & Truhlar, D. G. (2022) · Electronic Structure · doi:10.1088/2516-1075/ac94ac · Cited by 23
Low-cost multiclass-image encryption based on compressive sensing and chaotic system
Sha, Y., Mou, J., Banerjee, S., Jahanshahi, H., & Cao, Y. (2022) · Nonlinear Dynamics · doi:10.1007/s11071-022-08206-8 · Cited by 22
“Reconstruction process convolution kernel requires a large amount of storage space, especially at high sampling rate, the quality of signal reconstruction is lower than traditional ones [40].”
BackgroundChen, C., Jiang, B., Liu, S., Li, C., Wu, C., & Yin, R. (2023) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2023.3318080 · Cited by 26
Bhalgaonkar, S., Munot, M. V., & Anuse, A. D. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109180 · Cited by 29
McDermott, M. B. A., Hsu, T. M. H., Weng, W.-H., Ghassemi, M., & Szolovits, P. (2020) · arXiv · doi:10.48550/arxiv.2006.15229 · Cited by 22
“We believe CheXpert++ could also be optimized further, via neural network acceleration (e.g., Lebedev and Lempitsky (2018); Cheng et al. (2018)).”
BackgroundMethodWu, Y. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112188 · Cited by 27
Dynamic synchronous graph transformer network for region-level air-quality forecasting
Xia, H., Chen, X., Chen, B., & Hu, Y. H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128924 · Cited by 22
Hybrid Fully Connected Tensorized Compression Network for Hyperspectral Image Classification
Li, H.-C., Lin, Z., Ma, T.-Y., Zhao, X.-L., Plaza, A., & Emery, W. J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3241193 · Cited by 21
“Although the convolutional layers have been compressed by these compact structures, the fully connected (FC) layers still have a large number of parameters [45].”
BackgroundCompact Broad Learning System Based on Fused Lasso and Smooth Lasso
Chu, F., Liang, T., Chen, C. L. P., Wang, X., & Ma, X. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3267947 · Cited by 25
“However, the combination of many groups of nodes also increases the complexity of the model [11].”
BackgroundLearning to Explore Distillability and Sparsability: A Joint Framework for Model Compression
Liu, Y., Cao, J., Li, B., Hu, W., & Maybank, S. J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3185317 · Cited by 34
Lightweight Tensorized Neural Networks for Hyperspectral Image Classification
Ma, T.-Y., Li, H.-C., Wang, R., Du, Q., Jia, X., & Plaza, A. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3225438 · Cited by 20
“Although deep models have shown excellent performance in HSI classification, they are frequently overparameterized, which increases the demands for computing and storage resources (as well as the risk of overfitting), thus limiting their practical applications [42].”
BackgroundManzoor, H. U., Jafri, A. R., & Zoha, A. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101376 · Cited by 32
Self-Binarizing NetworksPreprint
Lahoud, F., Achanta, R., Márquez-Neila, P., & Susstrunk, S. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00730 · Cited by 30
“Not surprisingly, there is considerable interest in making deep learning models computationally efficient to better suit such devices (Ota et al., 2017; Cheng et al., 2018; Zhu, 2018).”
BackgroundHu, B., & Miao, H. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3502172 · Cited by 22
Gao, X., Wang, Y., Liu, B., Zhou, X., Zhang, R., Wang, J., Niyato, D. T., Kim, D. I., Jamalipour, A., Yuen, C., et al. (2026) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2026.3660854 · Cited by 44
“Finally, innovations such as tensor product attention and hierarchical memory systems o ff er targeted solutions for compacting complex multi-agent coordination modules [183].”
Efficient Visual Recognition: A Survey on Recent Advances and Brain-inspired Methodologies
Wu, Y., Wang, D., Lu, X., Yang, F., Yao, M., Dong, W., Shi, J., & Li, G. (2022) · Machine Intelligence Research · doi:10.1007/s11633-022-1340-5 · Cited by 25
“Deng et al. (2020) [11], one of the latest surveys, is comprehensive and professional in both DNN compression and hardware design.”
BackgroundInfluentialSelf-citationAutomated Detection of Multitype Landforms on Mars Using a Light-Weight Deep Learning-Based Detector
Jiang, S., Lian, Z., Yung, K. L., Ip, W. H., & Gao, M. Y. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3169454 · Cited by 20
Oikonomou, K. M., Kansizoglou, I., & Gasteratos, A. C. (2023) · Machines · doi:10.3390/machines11020162 · Cited by 20
A Review of Intrusion Detection for Railway Perimeter Using Deep Learning-Based Methods
Wang, J., Zhai, H., Yang, Y., Xu, N., Li, H., & Fu, D. (2024) · IEEE Access · doi:10.1109/access.2024.3510746 · Cited by 24
Context-Aware Human Activity and Smartphone Position-Mining with Motion Sensors
Gao, Z., Liu, D., Huang, K., & Huang, Y. (2019) · Remote Sensing · doi:10.3390/rs11212531 · Cited by 19
“In recent years, compression and acceleration for deep neural networks have become a valuable research topic [51].”
Emerging Paradigms of Neural Network PruningPreprint
Wang, H., Qin, C., Zhang, Y., & Fu, Y. (2021) · arXiv (Cornell University) · Cited by 23
Li, S., Liu, F., Peng, G., Cheng, F., Zhao, B., & Ji, M. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3214300 · Cited by 18
Knowledge distillation with insufficient training data for regression
Kang, M., & Kang, S. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108001 · Cited by 20
Jeong, K., Oh, H., Lee, Y., Seo, H., Cho, G., Jeong, J., Park, G., Choi, J., Seo, Y.-D., Jeong, J.-H., et al. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3461775 · Cited by 20
“To address this issue, research should focus on model compression techniques [176] and lightweight deep learning architecture [177] to reduce the computational demands of AI systems.”
Pan, Z., Li, Y., Guan, Z., Liang, M., Li, A., Wang, J., & Kou, F. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128672 · Cited by 31
Haar Wavelet Feature Compression for Quantized Graph Convolutional Networks
Eliasof, M., Bodner, B. J., & Treister, E. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3285874 · Cited by 23
“A plethora of quantization schemes tailored for different scenarios have been proposed, including post-training and quantization-aware training schemes, uniform and nonuniform schemes, etc.; see [12], [13], [18] and references therein.”
BackgroundMethodWan, Y., Fan, X., Xu, B., & He, Z. (2023) · Laser & Photonics Review · doi:10.1002/lpor.202201018 · Cited by 18
Decentralized Federated Learning With Adaptive Configuration for Heterogeneous Participants
Liao, Y., Xu, Y., Xu, H., Wang, L., Qian, C., & Qiao, C. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3335403 · Cited by 25
A Reference Architecture for Data-Driven Intelligent Public Transportation Systems
di Torrepadula, F. R., Martino, S. D., Mazzocca, N., & Sannino, P. (2024) · IEEE Open Journal of Intelligent Transportation Systems · doi:10.1109/ojits.2024.3441048 · Cited by 21
“Thus, further strategies to cope with these limitations should be investigated, such as model compression techniques, aiming at reducing models and fulfilling edge requirements [62].”
BackgroundAutomating Deep Neural Network Model Selection for Edge Inference
Lu, B., Yang, J., Chen, L. Y., & Ren, S. (2019) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi48466.2019.00035 · Cited by 24
“To achieve accuracies comparable to or even exceeding human levels, today’s machine learning models are becoming increasingly more complex, as evidenced by DNNs that can contain millions of or even more parameters [1], [3], [4].”
BackgroundMethodInfluentialHRCP : High-ratio channel pruning for real-time object detection on resource-limited platform
Chen, Y., Li, R., & Li, R. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.046 · Cited by 18
Agarwal, M., Gupta, S. K., & Biswas, K. K. (2023) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01768-4 · Cited by 22
“[8] have used a combination of pruning techniques such as quantization and binarization, matrix factorization, and knowledge distillation to compress the models.”
BackgroundMethodInfluentialZeng, J., Shen, X., Zhou, K., & Cao, L. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.12.020 · Cited by 19
Layer-wise Deep Neural Network Pruning via Iteratively Reweighted Optimization
Jiang, T., Yang, X., Shi, Y., & Wang, H. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682464 · Cited by 17
“Therefore, the adoption of DNN to the embedded devices becomes a trend especially for the real-time and accurate prediction services [4, 5].”
BackgroundThe feasibility and inevitability of stealth attacks
Tyukin, I., Higham, D. J., Bastounis, A., Woldegeorgis, E., & Gorban, A. N. (2023) · IMA Journal of Applied Mathematics · doi:10.1093/imamat/hxad027 · Cited by 24
Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning
Gupta, M., Camci, E., Keneta, V. R., Vaidyanathan, A., Kanodia, R., James, A., Foo, C.-S., Wu, M., & Lin, J. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00144 · Cited by 31
A Novel Deep Learning Model Compression Algorithm
Zhao, M., Li, M., Peng, S., & Li, J. (2022) · Electronics · doi:10.3390/electronics11071066 · Cited by 20
Artificial Intelligence Techniques and Health Literacy: A Systematic Review
Abeo, A. N. A., Armstrong, S., Scriney, M., & Goss, H. R. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100269 · Cited by 36
“60,61 In contrast, traditional ML methods, such as linear discriminant analysis and K-nearest neighbors, are easier to understand and require less computing power.”
Analysis on the Number of Linear Regions of Piecewise Linear Neural Networks
Hu, Q., Zhang, H., Gao, F., Xing, C., & An, J. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3028431 · Cited by 27
Cooperative Inference of DNNs Over Noisy Wireless Channels
Yun, S., Kang, J., Choi, S., & Kim, I. (2021) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2021.3092179 · Cited by 17
Riemannian Low-Rank Model Compression for Federated Learning With Over-the-Air Aggregation
Xue, Y., & Lau, V. K. N. (2023) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2023.3284381 · Cited by 18
“However, the resulting weight matrices of both network pruning and 1 regularization will be randomly sparse, making the datapath design very challenging [14].”
BackgroundAccelerated deep-learning-based process monitoring of microfluidic inkjet printing
Kim, S. J., Choi, E., Won, D. Y., Han, G., Han, G., An, K., & Kim, S. (2023) · CIRP journal of manufacturing science and technology · doi:10.1016/j.cirpj.2023.07.010 · Cited by 17
Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
Marbán, A., Becking, D., Wiedemann, S., & Samek, W. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00369 · Cited by 19
“Among existing network compression techniques, pruning and quantization are two popular and effective techniques to reduce the redundancy of deep neural networks [6].”
MethodStructured Sparsity of Convolutional Neural Networks via Nonconvex Sparse Group Regularization
Bui, K., Park, F., Zhang, S., Qi, Y., & Xin, J. X. (2021) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2020.529564 · Cited by 21
Compression and acceleration of convolution neural network: a Genetic Algorithm based approach
Agarwal, M., Gupta, S. K., Biswas, M., & Garg, D. (2022) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-022-03793-1 · Cited by 20
“For compressing the DL models, researchers have proposed several methods 1 3 which may find in Chen et al. (2018), Cheng et al. (2017, 2018), He et al. (2017) and its references.”
BackgroundMethodDeepCABAC: Context-adaptive binary arithmetic coding for deep neural network compressionPreprint
Wiedemann, S., Kirchhoffer, H., Matlage, S., Haase, P., Marbán, A., Marinč, T., Neumann, D., Osman, A., Marpe, D., Schwarz, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08318 · Cited by 25
“As a reaction, a plethora of work has been published on the topic of deep neural network compression (Cheng et al., 2017; Cheng et al., 2018).”
BackgroundChannel-Correlation-Based Selective Knowledge Distillation
Gou, J., Xiong, X., Yu, B., Zhan, Y., & Zhang, Y. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3232569 · Cited by 17
Synthetic data generation method for data-free knowledge distillation in regression neural networks
Zhou, T., & Chiam, K. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120327 · Cited by 28
Compact Neural Network via Stacking Hybrid Units
Lan, W., Cheung, Y., Jiang, J., Hu, Z., & Li, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3323496 · Cited by 16
“Amongthem,networkpruning hasbeenamainstreambranchinbothacademiaandindustry due to the advantages of saving resources, reducing latency and addressing privacy concerns, showing broad prospects in numerous applications [17], [18].”
BackgroundESM data downscaling: a comparison of super-resolution deep learning models
Pawar, N. M., Soltanmohammadi, R., Mahjour, S. K., & Faroughi, S. A. (2024) · Earth Science Informatics · doi:10.1007/s12145-024-01357-9 · Cited by 15
Unleashing the Low-Precision Computation Potential of Tensor Cores on GPUs
Li, G., Xue, J., Liu, L., Wang, X., Ma, X., Dong, X., Li, J., & Feng, X. (2021) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo51591.2021.9370335 · Cited by 25
“In addition, they often require developers to become heavily involved in tuning for reducing the quantization errors induced, by applying, e.g., re-training [15] and calibration techniques [17].”
BackgroundFault Detection of Li–Ion Batteries in Electric Vehicles: A Comprehensive Review
Li, H., Shaukat, H., Zhu, R., Kaleem, M. B., & Wu, Y. (2025) · Sustainability · doi:10.3390/su17146322 · Cited by 20
Optimized FPGA-based Deep Learning Accelerator for Sparse CNN using High Bandwidth Memory
Jiang, C., Ojika, D., Patel, B., & Lam, H. X. (2021) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm51124.2021.00026 · Cited by 23
“One of the main approaches to improve DNN performance is to exploit the high degree of redundancy in the weights parameters of DNN models [13]–[17].”
PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors
So, H. M., Bose, L., Dudek, P., & Wetzstein, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02384 · Cited by 22
“Broadly speaking, these efforts focus on developing smaller [34] or more efficient network architectures, often by pruning or quantizing larger models [19].”
Log-sum enhanced sparse deep neural network
Qiao, C., Shi, Y. H., Diao, Y.-X., Calhoun, V. D., & Wang, Y. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.04.118 · Cited by 17
WLAN interference signal recognition using an improved quadruple generative adversarial network
Xu, X., Jiang, T., Gong, J., Xu, H., & Qin, X. (2021) · Digital Signal Processing · doi:10.1016/j.dsp.2021.103188 · Cited by 14
Distributed Edge System Orchestration for Web-based Mobile Augmented Reality Services
Ren, P., Liu, L., Qiao, X., & Chen, J. (2022) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2022.3190375 · Cited by 21
“To meet the accurate and efficient requirements, researchers have recently been focusing on model compression and acceleration [20], [21], but these solutions are usually time-and resources consuming, making it difficult to provide timely services for AR applications.”
BackgroundMOSAIC: Spatially-Multiplexed Edge AI Optimization over Multiple Concurrent Video Sensing Streams
Gokarn, I., Sabbella, H., Hu, Y., Abdelzaher, T., & Misra, A. (2023) · ACM Multimedia Systems Conference (MMSys) · doi:10.1145/3587819.3590986 · Cited by 19
“Within the broader vision and AI community, a large body of work addresses the computational overhead of DNNs with techniques for early exit and model selection [7], model compression [9], weights quantization and weights/feature maps sparsification [31], and dynamic pruning [19].”
BackgroundObject Knowledge Distillation for Joint Detection and Tracking in Satellite Videos
Zhang, W., Deng, W., Cui, Z., Liu, J., & Jiao, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3355933 · Cited by 25
“To solve this difficulty, a large number of model compression and acceleration technologies [49], [50] have emerged, and KD [51] is one of them, which can effectively learn from large teacher models to small student models.”
BackgroundCompression of Deep Neural Networks by Combining Pruning and Low Rank Decomposition
Goyal, S., Choudhury, A. R., & Sharma, V. K. (2019) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw.2019.00162 · Cited by 16
“[6] also study model compression challenges and principles.”
BackgroundCompression of Convolutional Neural Network for Natural Language Processing
Wróbel, K., Karwatowski, M., Wielgosz, M., Pietroń, M., & Wiatr, K. (2020) · Computer Science · doi:10.7494/csci.2020.21.1.3375 · Cited by 14
Is Pruning Compression?: Investigating Pruning Via Network Layer Similarity
Blakeney, C., Yan, Y., & Zong, Z. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093318 · Cited by 18
Raza, S. M., Abidi, S. M. H., Masuduzzaman, M., & Shin, S. Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131357 · Cited by 13
Wang, Q., Yan, H., Yang, Y., Zhao, X., Huang, G., Huang, Z., Zhu, Z., Liu, S. V., Yi, B., Huang, G., et al. (2025) · Batteries · doi:10.3390/batteries11110414 · Cited by 14
“There is thus a challenge in distilling complex spatio-temporal models into lightweight forms suitable for the edge on board the vehicle [166].”
Sensitivity in Tensor Decomposition
Tichavský, P., Phan, A. H., & Cichocki, A. S. (2019) · IEEE Signal Processing Letters · doi:10.1109/lsp.2019.2943060 · Cited by 17
Dehouck, M., Anderson, M., & Gómez‐Rodríguez, C. (2020) · International Workshop/Conference on Parsing Technologies · doi:10.18653/v1/2020.iwpt-1.20 · Cited by 13
Symmetrical filters in convolutional neural networks
Dzhezyan, G., & Cecotti, H. (2019) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-021-01290-z · Cited by 13
“Model compression and acceleration in deep networks can be divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation [7].”
BackgroundMulti-Modal Data and Model Reduction for Enabling Edge Fusion in Connected Vehicle Environments
Thornton, S., & Dey, S. (2024) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2024.3378182 · Cited by 20
“Many of these improvements fall under the umbrella of model compression [58] [59].”
BackgroundSustainable transparency on recommender systems: Bayesian ranking of images for explainability
Paz-Ruza, J., Alonso‐Betanzos, A., Guijarro‐Berdiñas, B., Cancela, B., & Eiras‐Franco, C. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102497 · Cited by 14
“AI and Deep Learning models are known to consume significant computational resources, resulting in substantial carbon emissions and large model sizes; these environmental concerns highlight the need to strike a balance between model efficiency and explainability [20] [4].”
Boggust, A., Sivaraman, V., Assogba, Y., Ren, D., Moritz, D., & Hohman, F. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456371 · Cited by 14
“Model compression encompasses various techniques that reduce the storage space, memory, power, or time required to run an ML model while preserving its original behavior as much as possible [7,8,10,38, 57].”
BackgroundGuo, J., Wang, S., Ni, L. M.-S., & Shum, H. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300720 · Cited by 31
Photon-Driven Neural Reconstruction for Path Guiding
Zhu, S., Xu, Z., Sun, T., Kuznetsov, A. A., Meyer, M., Jensen, H. W., Su, H., & Ramamoorthi, R. (2021) · ACM Transactions on Graphics · doi:10.1145/3476828 · Cited by 20
“We believe our architecture can be further improved by advanced compression techniques [Cheng et al. 2018; Deng et al. 2020] and novel neural components, and we leave this as future work.”
MethodMusa, A., Hamada, M., & Hassan, M. (2022) · SHS Web of Conferences · doi:10.1051/shsconf/202213903002 · Cited by 8
Lu, Q., & Murmann, B. (2023) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3591466 · Cited by 25
Wiedemann, S., Mehari, T., Kepp, K., & Samek, W. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00368 · Cited by 27
“A lot of research is dedicated to improve the performance at inference time [6, 25, 28].”
BackgroundManzoor, H. U., Arshad, K., Assaleh, K. T., & Zoha, A. (2024) · doi:10.36227/techrxiv.171468119.91309905/v1 · Cited by 14
A Comprehensive Review of Machine-Learning Approaches for Crystal Structure/Property Prediction
Sadeghian, M., Palevičius, A., & Janušas, G. (2025) · Crystals · doi:10.3390/cryst15110925 · Cited by 21
Enabling Inference and Training of Deep Learning Models for AI Applications on IoT Edge Devices
Sharma, D., & Sarkar, S. (2022) · Internet of things · doi:10.1007/978-3-030-87059-1_10 · Cited by 19
Training Compact DNNs with ℓ 1 / 2 Regularization
Tang, A., Niu, L., Miao, J., & Zhang, P. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109206 · Cited by 12
Operating Critical Machine Learning Models in Resource Constrained Regimes
Selvan, R., Schön, J., & Dam, E. B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47425-5_29 · Cited by 14
“We do not explicitly study the influence of model selection using efficient neural architecture search [29] or model compression using pruning or similar methods [6], as these techniques require additional design choices outside of the normal model development and deployment phases.”
MethodSpike-and-Slab Shrinkage Priors for Structurally Sparse Bayesian Neural Networks
Jantre, S. R., Bhattacharya, S., & Maiti, T. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3485529 · Cited by 18
“…there exists no work that does a thorough theoretical and numerical investigation of shrinkage priors, such as Lasso and Horseshoe, in the context of variational BNNs. A. Related Work 1) Sparse Neural Networks: A number of approaches to neural network compression via sparsity include [12] and [13].”
Smart grid security through fusion-enhanced federated learning against adversarial attacks
Shabbir, A., Manzoor, H. U., Zoha, A., & Halim, Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111169 · Cited by 17
Tawiah, T. A.-Q. (2020) · International Journal of Advanced Robotic Systems · doi:10.1177/1729881420972278 · Cited by 16
“Different architectural configurations have also been realized to synthesize spatiotemporal features including residual block CNN and skipped connections configuration.(100) Recently, a connection between CNNs and multilayer CONV sparse coding(101) was established.”
BackgroundLogic Shrinkage: Learned FPGA Netlist Sparsity for Efficient Neural Network Inference
Wang, E., Davis, J. J., Stavrou, G.-I., Cheung, P. Y. K., Constantinides, G. A., & Abdelfattah, M. S. (2022) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3490422.3502360 · Cited by 20
“In order to exploit this in the quest for ever-greater performance within given area and power budgets, researchers and industrial practitioners alike are increasingly turning to lowprecision data types [1, 17, 22].”
BackgroundScenario Based Run-Time Switching for Adaptive CNN-Based Applications at the Edge
Minakova, S., Sapra, D., Stefanov, T. P., & Pimentel, A. D. (2022) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3488718 · Cited by 15
Complete vector quantization of feedforward neural networks
Floropoulos, N., & Tefas, A. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.08.003 · Cited by 12
Distillation of an End-to-End Oracle for Face Verification and Recognition Sensors
Guzzi, F., Bortoli, L. D., Molina, R., Marsi, S., Carrato, S., & Ramponi, G. (2020) · Sensors · doi:10.3390/s20051369 · Cited by 12
“A recent and detailed survey on the general principles of distillation and model compression is presented in [19].”
BackgroundEnhancing disruption prediction through Bayesian neural network in KSTAR
Kim, J., Lee, J., Seo, J., Ghim, Y.-C., Lee, Y., & Na, Y.-S. (2024) · Plasma Physics and Controlled Fusion · doi:10.1088/1361-6587/ad48b7 · Cited by 14
“This requires model acceleration techniques [33].”
Mohanty, L., Kumar, A., Mehta, V., Agarwal, M., & Suri, J. S. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19192-x · Cited by 14
Compressing Deep Image Super-Resolution Models
Jiang, Y., Nawała, J., Zhang, F., & Bull, D. (2024) · Picture Coding Symposium · doi:10.1109/pcs60826.2024.10566374 · Cited by 11
Fully Tensorized Lightweight ConvLSTM Neural Networks for Hyperspectral Image Classification
Ma, T.-Y., Li, H.-C., Zheng, Y., Du, Q., & Plaza, A. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3511575 · Cited by 15
“This heightened complexity raises the demands for computing and storage resources and can increase the danger of overfitting, thereby restricting their practical applications [35].”
Naser, M. Z. (2025) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2025.116231 · Cited by 16
Novel Arithmetics to Accelerate Machine Learning Classifiers in Autonomous Driving Applications
Cococcioni, M., Rossi, F., Ruffaldi, E., & Saponara, S. (2019) · IEEE International Conference on Electronics, Circuits and Systems (ICECS) · doi:10.1109/icecs46596.2019.8965031 · Cited by 11
Connected AR for Combating COVID-19
Amano, T., Yamaguchi, H., & Teruo Higashino, Y. (2020) · IEEE Internet of Things Magazine · doi:10.1109/iotm.0001.2000149 · Cited by 11
“Model compression approaches are promising to run DNNs in a resource-limited environment [13].”
BackgroundMo, H., Liu, L., Hu, W., Zhu, W., Li, Q., Li, A., Yin, S., Chen, J., Jiang, X., & Wei, S. (2020) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00067 · Cited by 12
Heuristic-based automatic pruning of deep neural networks
Choudhary, T., Mishra, V. K., Goswami, A., & Jagannathan, S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-021-06679-z · Cited by 11
“ation techniques, the readers are suggested to refer to recent surveys [8, 9].”
BackgroundSeNPIS: Sequential Network Pruning by class-wise Importance Score
Pachón, C. G., Ballesteros, D. M., & Renza, D. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109558 · Cited by 12
Metadata
- Authors
- Cheng, Y., Wang, D., Zhou, P., & Zhang, T.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- Harbin Institute of Technology (CN)
- Huazhong University of Science and Technology (CN)
- Tsinghua University (CN)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2018-01
- Volume
- 35
- Issue
- 1
- Pages
- 126-136
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W2783538964
- Cited by
- 817
- References
- 67
- Citations by year
- 2026: 114 · 2025: 132 · 2024: 116 · 2023: 110 · 2022: 109 · 2021: 92 · 2020: 89 · 2019: 41 · 2018: 14
- FWCI
- 29.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- model compression, deep neural network, inference acceleration, parameter reduction, convolutional neural networks, training efficiency, GPU computing, face verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.