Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges
The DOI matches its Crossref record; 61 of the 67 references it cites were found in a registry or an index.
Abstract
In recent years, deep neural networks (DNNs) have received increased attention, have been applied to different applications, and achieved dramatic accuracy improvements in many tasks. These works rely on deep networks with millions or even billions of parameters, and the availability of graphics processing units (GPUs) with very high computation capability plays a key role in their success. For example, Krizhevsky et al. achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully connected layers. Usually, it takes two to three days to train the whole model on the ImagetNet data set with an NVIDIA K40 machine. In another example, the top face-verification results from the Labeled Faces in the Wild (LFW) data set were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally connected, and fully connected layers. It is also very time-consuming to train such a model to obtain a reasonable performance. In architectures that only rely on fully connected layers, the number of parameters can grow to billions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“but there is no review discussing all three categories which in future will be integrated seamlessly to design effective and efficient DNNs in the new edge intelligence era.”
From Bringing ai to edge: from deep learning’s perspective (2022) · cites this paper as [26]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Recently, more state-of-the-art architectures are used as baseline models in many works, including network in networks [67], VGGNets [68], and ResNets [69].”
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“[1] achieved breakthrough results in the 2012 ImageNet Challenge using a network containing 60 million parameters with five convolutional layers and three fully-connected layers.”
MethodInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“The work in [51] introduced a KD compression framework, which eased the training of deep networks by following a student-teacher paradigm, in which the student was penalized according to a softened version of the teacher’s output.”
BackgroundDeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · cited 1×
“example is the top face verification results on the Labeled Faces in the Wild (LFW) dataset were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally-connected, and fully-connected layers [2], [3].”
MethodHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_32 · Cited by 4,408 · cited 2×
“works that directly train CNNs with binary weights, for instance, BinaryConnect [13], BinaryNet [14] and XNORNetworks [15].”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 2×
“Other popular standard models include LeNets [66], All-CNN-nets [67] and many others.”
Method
Cited by
Items 301–400 of 818
Large deviation analysis of function sensitivity in random deep neural networks
Li, B., & Saad, D. (2020) · Aston Publications Explorer (Aston University) · Cited by 11
Beyond Human-Level Accuracy: Computational Challenges in Deep LearningPreprint
Hestness, J., Ardalani, N., & Diamos, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01736 · Cited by 22
“ts, but some algorithmic optimizations may be promising to chip away at this gap. Model compression or distillation, and low-precision or sparse computation may reduce model or activation tensor size [9], and reduce memory requirements by 1:5–10. Currently, many challenges exist to use these techniques during model tr…”
BackgroundOptimizing deep learning models for on-orbit deployment through neural architecture search
Prete, R. D., Thind, P. K., Mazzeo, A., Whitley, M., Papa, L., Longépé, N., & Meoni, G. (2025) · Scientific Reports · doi:10.1038/s41598-025-21467-8 · Cited by 14
“Despite its storage and computational efficiency, quantization requires careful calibration to balance bitwidth reduction with minimal accuracy degradation due to inherent trade-offs 37,38 .”
Lee, D. H., Park, E. G., Lee, Y. J., Jeong, H.-S., Roh, H.-Y., Jeong, G.-R., Kim, S.-W., & Kim, H. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms262210925 · Cited by 16
Hardware-friendly compression and hardware acceleration for transformer: A survey
Huang, S., Tang, E., Li, S., Ping, X., & Chen, R. (2022) · Electronic Research Archive · doi:10.3934/era.2022192 · Cited by 16
QoS-Ensured Model Optimization for AIoT: A Multi-Scale Reinforcement Learning Approach
Wu, G., Zhou, F., Qu, Y., Luo, P., & Li, X. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3294512 · Cited by 13
“Therefore, on one hand, more effective channel combinations are ignored (e.g., reducing the channel for some channels while increasing for others) [43].”
BackgroundInference serving with end-to-end latency SLOs over dynamic edge networks
Nigade, V., Bauszat, P., Bal, H. E., & Wang, L. (2024) · Real-Time Systems · doi:10.1007/s11241-024-09418-4 · Cited by 16
Keyhole Porosity Identification and Localization via X-Ray Imaging With YOLO
Ansari, M. A., Crampton, A. S., & Parkinson, S. (2024) · IEEE Access · doi:10.1109/access.2024.3393128 · Cited by 12
Uncertainty-based knowledge distillation for Bayesian deep neural network compression
Hemmatian, M., Shahzadi, A., & Mozaffari, S. (2024) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2024.109301 · Cited by 15
Yuan, H. (2025) · Medicine Advances · doi:10.1002/med4.70001 · Cited by 12
“Among common techniques, parameter pruning and low‐rank factorization have been suggested because they can be easily deployed in both the training and inference stages [13].”
Towards Low-Bit Quantization of Deep Neural Networks with Limited Data
Yuan, Y., Chen, C., Hu, X., & Peng, S. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412678 · Cited by 12
“However, the significant increase of the computational complexity hinders the deployment of DNNs on embedded devices [1] [2].”
BackgroundExploring lottery ticket hypothesis in media recommender systems
Wang, Y., Sui, Y., Wang, X., Liu, Z., & He, X. (2022) · International Journal of Intelligent Systems · doi:10.1002/int.22827 · Cited by 17
Residual-guided In-loop Filter Using Convolution Neural Network
Jia, W., Li, L., Li, Z., Zhang, X., & Liu, S. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3460820 · Cited by 12
Hammer, J., Schirrmeister, R. T., Hartmann, K., Marusič, P., Schulze‐Bonhage, A., & Ball, T. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac6770 · Cited by 13
Priority Neuron: A Resource-Aware Neural Network for Cyber-Physical Systems
Amir, M., & Givargis, T. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2857319 · Cited by 15
“Many network pruning and model reduction 217 techniques are proposed in the previous work with promising 218 results [28]–[30].”
MethodBuffer Sizes Reduction for Memory-efficient CNN Inference on Mobile and Embedded Devices
Minakova, S., & Stefanov, T. P. (2020) · Euromicro Conference on Digital System Design (DSD) · doi:10.1109/dsd51259.2020.00031 · Cited by 10
Neural Network Compression for Noisy Storage Devices
Isik, B., Choi, K., Zheng, X., Weissman, T., Ermon, S., Wong, H. ‐. P., & Alaghi, A. (2023) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3588436 · Cited by 14
“Additionally, we assume that each weight is a sample from a normal distribution W ij ∼ N (0 , 1) (Dziugaite et al., 2018; Zhou et al., 2018).”
BackgroundNousias, S., Pikoulis, E. V., Mavrokefalidis, C., & Lalos, A. S. (2021) · IEEE International Conference on Industrial Cyber-Physical Systems (ICPS) · doi:10.1109/icps49255.2021.9468126 · Cited by 11
Zou, W., Qi, X., Wu, Z., Wang, Z., Sun, M., & Shan, C. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669551 · Cited by 15
Enforcement Heuristics for Argumentation with Deep Reinforcement Learning
Craandijk, D., & Bex, F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20497 · Cited by 16
“Implementing methods for speeding up deep learning models constitutes an active research area (Cheng et al. 2018) and falls beyond the scope of this research.”
MethodKnowledge distillation: A novel approach for deep feature selection
Deepa, C., Shetty, A., & Narasimhadhan, A. V. (2022) · The Egyptian Journal of Remote Sensing and Space Science · doi:10.1016/j.ejrs.2022.12.006 · Cited by 10
IPAD: Iterative pruning with activation deviation for sclera biometrics
Vitek, M., Bizjak, M., Peer, P., & Štruc, V. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101630 · Cited by 15
“For a more in-depth overview of the existing literature, we refer the reader to Cheng et al. (2018), Liang et al. (2021), Choudhary et al. (2020) for comprehensive review papers on model compression and to Nigam et al. (2015); Das et al. (2013); Vitek et al. (2023) for surveys and comparative…”
Slim-Tasnet: A Slimmable Neural Network for Speech Separation
Elminshawi, M., Chetupalli, S. R., & Habets, E. A. P. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248143 · Cited by 12
Jiang, J., Zhou, K., Liang, J., Tian, F., Zhao, C., Yang, J., Xue, X., & Zeng, X. (2023) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2023.3321107 · Cited by 12
“Quantization and pruning are common approaches for weight compression of deep networks but often at the cost of system performance [1].”
BackgroundLi, Q., Li, H., & Meng, L. (2024) · Journal of Real-Time Image Processing · doi:10.1007/s11554-024-01496-8 · Cited by 11
“They take up most of the FLOPs [18], therefore are the main compression target of channel pruning.”
Failure-Resilient Distributed Inference With Model Compression Over Heterogeneous Edge Devices
Wang, L., Li, L., Xu, L., Peng, X., & Fei, A. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3419831 · Cited by 13
“ToalleviatethecomputationburdenofedgedevicesforDNN inference,somemodelcompressiontechniquesexistinthe literature,whereinKnowledgeDistillation(KD)standsoutas aneffectivesolutionforproducingamorecompactmodelthat canserveasasubstituteforacomplexmodel[1].”
Liang, X., Shim, J., & Song, D. (2025) · Building and Environment · doi:10.1016/j.buildenv.2025.112666 · Cited by 11
An adaptive dual distillation framework for efficient remaining useful life prediction
Cheng, X., Chaw, J. K., Sahrani, S., Ang, M. C., Gunasekaran, S. S., Mahmoud, M. A., Zaman, H. B., Zhao, Y., & Ren, F. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01886-w · Cited by 11
“Research identifies four main methods for building lightweight networks for RUL prediction: weight pruning, quantization, knowledge distillation, and NAS (neural architecture search) [9].”
Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
Sathyam, R., & Li, Y. (2025) · IEEE Open Journal of Vehicular Technology · doi:10.1109/ojvt.2025.3604823 · Cited by 20
Rivera, E. F., Guerrero-Vásquez, L. F., & Robles-Bykbaev, V. (2026) · Technologies · doi:10.3390/technologies14010076 · Cited by 18
Deep network compression with teacher latent subspace learning and LASSO
Oyedotun, O. K., Shabayek, A. E. R., Aouada, D., & Ottersten, B. (2020) · Applied Intelligence · doi:10.1007/s10489-020-01858-2 · Cited by 11
“Unfortunately, low-rank factorization methods are computationally expensive, and require extensive model retraining after compression [38].”
BackgroundMethodPs and Qs: Quantization-Aware Pruning for Efficient Low Latency Neural Network Inference.
Hawks, B., Duarte, J. M., Fraser, N. J., Pappalardo, A., Tran, N. V., & Umuroglu, Y. (2021) · eScholarship (California Digital Library) · Cited by 9
Implementation of CT Image Segmentation Based on an Image Segmentation Algorithm
Shen, L. (2022) · Applied Bionics and Biomechanics · doi:10.1155/2022/2047537 · Cited by 9
“However, it has low sensitivity, long image acquisition time, and moderate price [14].”
RetractedBeta network for boundary detection under nondeterministic labels
Li, M., Chen, D., & Liu, S. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110389 · Cited by 10
Sparsity-Aware Orthogonal Initialization of Deep Neural Networks
Esguerra, K., Nasir, M., Tang, T. B., Tumian, A., & Ho, E. T. W. (2023) · IEEE Access · doi:10.1109/access.2023.3295344 · Cited by 14
“Model compression techniques are methods to reduce these costs without degrading the model performance [7].”
MethodSQUAT: Stateful Quantization-Aware Training in Recurrent Spiking Neural Networks
Venkatesh, S., Marinescu, R. V., & Eshraghian, J. K. (2024) · Neuro Inspired Computational Elements Conference (NICE) · doi:10.1109/nice61972.2024.10549198 · Cited by 19
“Techniques such as pruning [4], specialized data encoding [5], model compression [6], early exiting [7], amongst many others, can be used to reduce the computational cost of running a neural network [8], [9].”
MethodAn Investigation on Open-RAN Specifications: Use Cases, Security Threats, Requirements, Discussions
Park, H., Nguyen, T., & Park, L. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.052394 · Cited by 22
Large deviation analysis of function sensitivity in random deep neural networks
Li, B., & Saad, D. (2020) · Journal of Physics A Mathematical and Theoretical · doi:10.1088/1751-8121/ab6a6f · Cited by 13
Reduced-Order Modeling of Deep Neural Networks
Gusak, J., Daulbaev, T., Ponomarev, E., Cichocki, A. S., & Oseledets, I. V. (2021) · Computational Mathematics and Mathematical Physics · doi:10.1134/s0965542521050109 · Cited by 11
Model Compression via Collaborative Data-Free Knowledge Distillation for Edge Intelligence
Hao, Z., Luo, Y., Wang, Z., Hu, H., & An, J. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428308 · Cited by 9
“Pruning Han Hu is the corresponding author. explores the redundancy in model weights, and removes redundant and noncritical ones [1].”
BackgroundMemory-Throughput Trade-off for CNN-Based Applications at the Edge
Minakova, S., & Stefanov, T. P. (2022) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3527457 · Cited by 14
Broad Minimax Probability Learning System and its Application in Regression Modeling
Chu, F., Liang, T., Chen, C. L. P., Ma, X., & Wang, X. (2022) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2022.3211519 · Cited by 11
“D EEP structure networks [1], [2] are powerful at large scale data processing, but they suffer from the time-consuming training process and expensive computing hardware [3].”
BackgroundFifthNet: Structured Compact Neural Networks for Automatic Chord Recognition
O’Hanlon, K., & Sandler, M. B. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3070158 · Cited by 13
Intelligent Radio Signal Processing: A Contemporary Survey.Preprint
Pham, Q., Nguyen, N. T., Huynh‐The, T., Le, L. B., Lee, K., & Hwang, W. (2020) · arXiv (Cornell University) · Cited by 9
Scaling Up Exact Neural Network Compression by ReLU StabilityPreprint
Serra, T., Yu, X., Kumar, A., & Ramalingam, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07804 · Cited by 32
An Attention-Based Deep Generative Model for Anomaly Detection in Industrial Control Systems
Macas, M., Wu, C., & Fuertes, W. (2023) · International Conference on Web Information Systems and Technologies · doi:10.5220/0012264000003584 · Cited by 11
“…(ii) Using more lightweight models, such as SqueezeNet (Iandola et al., 2016), or employing other techniques to compress deep neural networks (Cheng et al., 2018) could facilitate deployment over resource-constrained devices; (iii) Investigating better windowing strategies could improve the…”
MethodMhapsekar, R. U., Abraham, L., Davy, S., & Dey, I. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3475634 · Cited by 10
Enhancing Binary Convolutional Neural Networks for Hyperspectral Image Classification
Tang, X., Zhang, K., Zhou, X., Zeng, L., & Huang, S. (2024) · Remote Sensing · doi:10.3390/rs16234398 · Cited by 8
Relation-Guided Adversarial Learning for Data-Free Knowledge Transfer
Liang, Y., & Fu, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02303-4 · Cited by 12
“However, these well-designed models require significant computational resources, which limits their deployment on resource-limited platforms such as mobile phones and edge devices (Cheng et al, 2018).”
Obaid, L., Hamad, K., Al-Ruzouq, R. I., Dabous, S. A., & Ismail, K. (2025) · Green Energy and Intelligent Transportation · doi:10.1016/j.geits.2025.100340 · Cited by 14
Multimodal Sensing for Intelligent V2X: A Review of Recent Advances Toward Deployment
Tan, K., & Zhu, C. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3601236 · Cited by 22
“Another example is different model compression techniques containing pruning, quantization, knowledge distillation, etc. [154], [155].”
A Power-Efficient Parameter Quantization Technique for CNN Accelerators
Kalalı, E., & van Leuken, R. (2021) · Euromicro Conference on Digital System Design (DSD) · doi:10.1109/dsd53832.2021.00012 · Cited by 8
Gong, Y., & Zhou, J. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/9770747 · Cited by 8
“Different environments and different projects may have different data mining processes [10].”
BackgroundRetractedSen, T., & Shen, H. (2022) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs54860.2022.00062 · Cited by 12
AttTrack: Online Deep Attention Transfer for Multi-object Tracking
Nalaie, K., & Zheng, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00170 · Cited by 9
“Extensive parameter tuning is required for network pruning to work without significant loss of accuracy [5].”
BackgroundTalaria: Interactively Optimizing Machine Learning Models for Efficient Inference
Hohman, F., Wang, C., Lee, J., Görtler, J., Moritz, D., Bigham, J. P., Ren, Z., Foret, C., Shan, Q., & Zhang, X. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642628 · Cited by 15
“While surveys detail different compression techniques [17, 20, 23], most existing practical guidance stems from online tutorials and documentation from popular ML libraries.”
A survey on GAN acceleration using memory compression techniques
Tantawy, D., Zahran, M., & Wassal, A. G. (2021) · Journal of Engineering and Applied Science · doi:10.1186/s44147-021-00045-5 · Cited by 10
A Low-Complexity Modified ThiNet Algorithm for Pruning Convolutional Neural Networks
Tofigh, S., Ahmad, M. O., & Swamy, M. N. S. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3164328 · Cited by 11
Zhou, Y. Z., Xia, L., Zhao, J., Yao, R., & Liu, B. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15608-2 · Cited by 15
A CHARACTERIZATION OF THE EDGE OF CRITICALITY IN BINARY ECHO STATE NETWORKS
Verzelli, P., Livi, L., & Alippi, C. (2018) · Machine learning for signal processing ... · doi:10.1109/mlsp.2018.8516959 · Cited by 8
“This work sets also in the context of reducing model complexity, in which binarization plays an important role both for theoretical aspects [1] and applied perspectives [2], since it would considerably reduce required hardware resources and speed-up training algorithms.”
BackgroundCompressed Superposition of Neural Networks for Deep Learning in Edge Computing
Zeman, M., Osipov, E., & Bosnić, Z. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533602 · Cited by 9
“In this way of pruning, L 1 and L 2 norm regularizers are normally used [5].”
BackgroundMethodInfluentialRim, D., Venturi, L., Bruna, J., & Peherstorfer, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13977 · Cited by 11
“The RDN is reminiscent of the compression framework for deep networks that is being studied theoretically for improving generalization bounds [36, 2], or being utilized in practice to accelerate the performance of large networks in practical applications [8, 38, 9].”
BackgroundLSTM-Based Model Compression for CAN Security in Intelligent Vehicles
Feng, Y., Lai, Y., Chen, Y., Zhang, Z., & Wei, J. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3438110 · Cited by 8
Efficient Log-based Anomaly Detection with Knowledge Distillation
Nguyen, H., Nguyen, L.-V., Le, V.-H., Zhang, H., & Le, M.-T. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00078 · Cited by 12
“Model compression is an important research topic for the deployment of DL models on low-end embedded GPU or CPU systems with limited resources [50].”
BackgroundMulti-Class Classification of Human Activity and Gait Events Using Heterogeneous Sensors
Javed, T., Raza, A., Maqbool, H. F., Zafar, S., Taborri, J., & Rossi, S. (2024) · Journal of Sensor and Actuator Networks · doi:10.3390/jsan13060085 · Cited by 9
Youm, S., & Go, S. (2025) · Applied Sciences · doi:10.3390/app15020890 · Cited by 11
Wang, G., Chen, H., Li, J., Wang, J., Liu, W., & Chen, L. (2025) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2025.3558725 · Cited by 8
Yang, K., Hu, B., & Zhao, M. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3574718 · Cited by 8
“Model compression and quantization (e.g., pruning or low-bit quantization) alleviate the computational load on weaker devices [16], [17].”
Park, W., Song, H., Kim, E., Choi, M. G., Lee, M. G., Rhee, H., Kim, G., Go, T., Martinez, A., Kim, D., et al. (2025) · Advanced Materials · doi:10.1002/adma.202502255 · Cited by 13
Li, A., Xu, Z., Pan, Y. A., Gao, B., Zhang, J., Chen, Y., & Li, Y. (2025) · Journal of Transportation Engineering Part A Systems · doi:10.1061/jtepbs.teeng-9105 · Cited by 9
Kundurthy, O. H., Kaata, S. K., Vikram, S., Somayajula, R., & Gangavarapu, R. (2025) · International Conference on Electronics and Computing, Communication Networking Automation Technologies (ICEC2NT) · doi:10.1109/icec2nt65402.2025.11380015 · Cited by 7
Yang, Z. Q., Zhai, Y., Xiang, Y., Wu, J., Shi, J., & Wu, Y. (2022) · IEEE Access · doi:10.1109/access.2022.3188119 · Cited by 8
“A generally accepted conclusion is that deep network models are severely overparameterized [27].”
BackgroundHan, G., Guo, W., Zhang, H., Jin, J., Gan, X., & Zhao, X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108489 · Cited by 9
Efficient Deep Learning in Network Compression and Acceleration
Ge, S. (2018) · IntechOpen eBooks · doi:10.5772/intechopen.79562 · Cited by 8
“Following these observations, many efficient methods for network compression and acceleration have been proposed, and several survey papers could be found in [12–14 ].”
MethodClassification of Multi-Frame Human Motion Using CNN-based Skeleton Extraction
Yoo, H., & Chung, K. (2022) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2022.024890 · Cited by 10
Space of Functions Computed by Deep-Layered Machines
Mozeika, A., Li, B., & Saad, D. (2020) · Physical Review Letters · doi:10.1103/physrevlett.125.168301 · Cited by 7
Gkalelis, N., & Mezaris, V. (2020) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2020.00028 · Cited by 11
“DNN compression and acceleration approaches can be roughly categorized to quantization, low-rank approximations, knowledge distillation and pruning [1], [2], [14].”
BackgroundPorn Streamer Recognition in Live Video Based on Multimodal Knowledge Distillation
Wang, L., Zhang, J., Yao, J., & Li, Z. (2021) · Chinese Journal of Electronics · doi:10.1049/cje.2021.07.027 · Cited by 7
Transformed $\ell_1$ Regularization for Learning Sparse Deep Neural NetworksPreprint
Ma, R., Miao, J., Niu, L., & Zhang, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.01021 · Cited by 7
DARC: Differentiable ARchitecture CompressionPreprint
Singh, S. K., Khetan, A., & Karnin, Z. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08170 · Cited by 8
“See Cheng et al. [34] for a survey of common approaches.”
Refactoring Neural Networks for VerificationPreprint
Shriver, D., Xu, D., Elbaum, S. G., & Dwyer, M. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08026 · Cited by 16
“, [29], [30], [31].”
Rate-Distortion Theoretic Model Compression: Successive Refinement for PruningPreprint
Isik, B., No, A., & Weissman, T. (2021) · arXiv (Cornell University) · Cited by 8
Kulkarni, U., Meena, S. M., & Gurlahosur, S. V. (2025) · SN Computer Science · doi:10.1007/s42979-025-03768-x · Cited by 6
“Considering an example MobileNet, a 53-layer architecture [5],it has more than 3.5 million parameters and needs more than 0.4 Giga Floating Point Operations (GFLOPs) [6] when inferencing an image of resolution 224 × 224 pixels.”
InfluentialAbdalla, A., Mohammed, M. M. A., Adedeji, O., Dotray, P. A., & Guo, W. (2025) · Smart Agricultural Technology · doi:10.1016/j.atech.2025.101086 · Cited by 6
Marmouzi, O., Oumaira, I., & Khaddar, M. A. E. (2026) · Sustainability · doi:10.3390/su18084115 · Cited by 11
Leveraging different learning styles for improved knowledge distillation in biomedical imaging
Niyaz, U., Sambyal, A. S., & Bathula, D. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107764 · Cited by 6
“Quantization-based approaches reduce the precision of numbers used to represent the neural network weights, and Low-rank approximation uses tensor decomposition to estimate the informative parameters of a network [8].”
MethodDistilling dynamical knowledge from stochastic reaction networks
Liu, C., & Wang, J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2317422121 · Cited by 14
Q-Net Compressor: Adaptive Quantization for Deep Learning on Resource-Constrained Devices
Singh, M. P., Mohanty, L., Gupta, N., Bansal, Y., & Garg, S. (2024) · International Conference on Computing, Sciences and Communications (ICCSC) · doi:10.1109/iccsc62048.2024.10830393 · Cited by 6
Ghosh, R. (2021) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-021-00518-2 · Cited by 7
LNNet: Lightweight Nested Network for motion deblurring
Guo, C., Wang, Q., Dai, H., Wang, H. H., & Li, P. (2022) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2022.102584 · Cited by 6
Research on Human Action Feature Detection and Recognition Algorithm Based on Deep Learning
Wu, Z., & Du, H. (2022) · Mobile Information Systems · doi:10.1155/2022/4652946 · Cited by 6
“+e network introduces a deconvolution structure to generate a mask map that is the same size as the input feature map, thereby generating a mask value classification for each pixel, and implementing instance segmentation for each pixel [20].”
BackgroundChen, Y., Hsu, C.-F., Tsai, C.-C., & Hsu, C.-H. (2022) · doi:10.1145/3552487.3556438 · Cited by 6
“Distillation was initially proposed to compress neural networks by transferring knowledge from a complex, teacher, model to a simple, student, model [10, 18, 23, 60].”
BackgroundResource Efficient Federated Learning for Deep Anomaly Detection in Industrial IoT applications
Γκίλλας, Α., & Lalos, A. S. (2023) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/dsp58604.2023.10167873 · Cited by 6
“Furthermore, inspired by the compression and acceleration techniques aiming to reduce the size of deep learning networks [16]–[19] and considering the limitations of the IIoT edge devices in terms of computational and power resources [1], [20], [21], a novel compression-based optimization problem is proposed at the se…”
BackgroundCompressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems
Lee, S., & Lee, J. (2019) · Applied Sciences · doi:10.3390/app9081669 · Cited by 8
“etwork architecture using reinforcement learning (e.g., NAS [37] and NASNet [38]), and tuning parameters that trade-off model size and prediction accuracy automatically [39]. We refer to Cheng et al. [40] for a more detailed survey of DNN model compression. 1.2. Contribution Even though there are quite a few works on…”
InfluentialOptimization Strategies in Quantized Neural Networks: A Review
Lin, Y., Lei, M., & Niu, L. (2019) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2019.00063 · Cited by 6
“More explicit introduction can be found in [16].”
BackgroundFactorized WaveNet for voice conversion with limited data
Du, H., Tian, X., Xie, L., & Li, H. (2021) · Speech Communication · doi:10.1016/j.specom.2021.03.003 · Cited by 7
Ozerov, A., & Duong, N. Q. K. (2021) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco54536.2021.9616244 · Cited by 9
“One way to solve this problem consists in using model compression techniques targeting at the same time to reduce the model size and to accelerate it at the inference [5].”
MethodTransferring Inter-Class Correlation for Teacher–Student frameworks with flexible models
Wen, H., Wu, Y., Li, J., Yang, C., Duan, H., & Yang, Y. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108316 · Cited by 6
A New Clustering-Based Technique for the Acceleration of Deep Convolutional Networks
Pikoulis, E. V., Mavrokefalidis, C., Nousias, S., & Lalos, A. S. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-981-16-3357-7_5 · Cited by 7
Automated Customization of On-Device Inference for Quality-of-Experience Enhancement
Bai, Y., Chen, L., Ren, S., & Xu, J. (2022) · IEEE Transactions on Computers · doi:10.1109/tc.2022.3208207 · Cited by 8
“Various mechanisms, e.g., model selection [5], DNN compression [6], and hyperparameter optimization [7]) have been studied to improve the performance of on-device inferences in terms of accuracy, inference delay, energy consumption, etc.”
BackgroundMethod
Metadata
- Authors
- Cheng, Y., Wang, D., Zhou, P., & Zhang, T.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- Harbin Institute of Technology (CN)
- Huazhong University of Science and Technology (CN)
- Tsinghua University (CN)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2018-01
- Volume
- 35
- Issue
- 1
- Pages
- 126-136
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W2783538964
- Cited by
- 817
- References
- 67
- Citations by year
- 2026: 114 · 2025: 132 · 2024: 116 · 2023: 110 · 2022: 109 · 2021: 92 · 2020: 89 · 2019: 41 · 2018: 14
- FWCI
- 29.51
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- model compression, deep neural network, inference acceleration, parameter reduction, convolutional neural networks, training efficiency, GPU computing, face verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.