PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index; 8 of the 13 sources its claims cite were checked for support.
Abstract
As edge devices become prevalent, deploying Deep Neural Networks (DNN) on edge devices has become a critical issue. However, DNN requires a high computational resource which is rarely available for edge devices. To handle this, we propose a novel model compression method for the devices with limited computational resources, called PQK consisting of pruning, quantization, and knowledge distillation (KD) processes. Unlike traditional pruning and KD, PQK makes use of unimportant weights pruned in the pruning process to make a teacher network for training a better student network without pre-training the teacher model. PQK has two phases. Phase 1 exploits iterative pruning and quantization-aware training to make a lightweight and power-efficient model. In phase 2, we make a teacher network by adding unimportant weights unused in phase 1 to a pruned network. By using this teacher network, we train the pruned network as a student network. In doing so, we do not need a pre-trained teacher network for the KD framework because the teacher and the student networks coexist within the same network. We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.
Overview
CoChat Summary
TL;DR
PQK is a model-compression framework that combines iterative pruning, quantization-aware training, and knowledge distillation by reusing pruned unimportant weights to form a teacher network without pre-training a separate teacher. It is evaluated on keyword spotting and image recognition, where phase 2 KD improves pruned/quantized student accuracy over phase 1.
Key findings
- On Google Speech Commands with ResNet-8-narrow at target pruning ratio \(p_t=0.9\), phase 2 improves the 4-bit pruned net from 74.1% to 83.1% and the 8-bit pruned net from 81.7% to 86.4%, gains of 9.0% and 4.7% respectively.
- On ResNet-8 at \(p_t=0.9\), phase 2 improves 8-bit pruned net accuracy from 92.6% to 94.0% and 4-bit from 91.7% to 93.6%; phase2-F reaches 94.4% at 8-bit versus 94.3% vanilla.
- On CIFAR100 with ResNet-32, phase2-F with an 8-bit pruned net reaches 71.1% versus 69.7% vanilla, outperforming vanilla by 1.4%; phase2-P reaches 69.8%.
- Ablation on ResNet-8-narrow 4-bit \(p_t=0.9\): phase2-P reaches 83.1%, outperforming the best finetuning baseline (lr=0.1, 79.2%) by 3.9%.
Methods
- PQK has two phases: phase 1 uses iterative magnitude-based unstructured pruning with a gradually increasing pruning ratio from \(p_i=0\) to target \(p_t\), updating the mask every 32 iterations, plus uniform symmetric per-layer quantization-aware training with a learnable step-size and straight-through estimator; phase 2 builds a full-net teacher by adding unimportant weights to the pruned student and trains both with KD using KL and cross-entropy losses, with a warm-up stage (\(\alpha=1,\beta=0\)) before setting \(T=2,\alpha=0.5,\beta=0.5\).
- Experiments use Google Speech Commands Dataset v1 with ResNet-8 and ResNet-8-narrow, and CIFAR100 with ResNet-32; target pruning ratio is 0.9 for KWS, with additional ResNet-8 runs at \(p_t\in\{0.9,0.7,0.5\}\), and quantization is 8-bit and 4-bit against a 32-bit baseline.
- Phase 2 adds 9 epochs for KWS and 60 epochs for CIFAR100; evaluation compares vanilla, phase1-P, phase2-P, and phase2-F test accuracy, plus an ablation against finetuning with the same training budget.
Limitations
- The paper states no limitations.
- Evident from the text: hyperparameters were not grid-searched but chosen from related works, and evaluation is limited to two tasks/datasets and specific ResNet models.
- Evident from the text: phase 2 requires additional training epochs, two forward passes, and different batchnorm parameters for pruned and full nets; the full net remains 32-bit while sharing the quantized pruned net, and combining 32-bit with 4-bit trained models is noted as more unstable than with 8-bit.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 3 Not valid · 0 Not checked · 28 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Concerning bitwidth and accuracy, 8-bit consistently outperforms 4-bit because of its high representative power from more bitwidth.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, there is a large performance gap between phase 1 and 2 in the pruned model of ResNet-8-narrow compared to that of ResNet-8.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In ResNet-8 with 4-bit, although pruned net performs well, accuracies of phase2-F are lower than those of phase2-P.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table ( p t = 0 . 9 ), the performance of phase1P decreases compared to vanilla.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
15 more claims
Create a free account to see all 35 claims and to verify the 4 references still open.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 8
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1. Experimental Setup · p. 3
“We set the warm up iteration ( s ) as 1500.”
§ Algorithm 1 PQK · p. 2
“13: for Epoch = 1 ,..., P 2 do 14: if s < Epoch then 15: set α, β // KD training after warm up stage”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“Image Recognition We use CIFAR100 Dataset [23] and choose ResNet-32 [24] for the baseline.”
MethodLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Feature maps [8, 9, 10] and logits of a network [11, 12] are widely used as knowledge.”
BackgroundEstimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910 · cited 4×
“Dequantization step just brings the quantized value back to the original range by multiplying the step-size: w̄ = ŵ × Sw. (3) These quantization and dequantization processes are nondifferentiable, so we utilize a straight-through estimator (STE) [19] for backpropagation.”
MethodInfluentialHubara, I., Soudry, D., & Yaniv, R. E. (2016) · arXiv · doi:10.48550/arxiv.1602.02505 · Cited by 1,667
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
He, Y., Liu, P., Wang, Z., Hu, Z., & Yang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00447 · Cited by 1,217 · cited 1×
“(1) Pruning method prunes the unimportant weights or channels based on different criteria [1, 2, 3, 4].”
MethodQuantizing deep convolutional networks for efficient inference: A whitepaper
Krishnamoorthi, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08342 · Cited by 753
A Comprehensive Overhaul of Feature Distillation
Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., & Choi, J. Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00201 · Cited by 706 · cited 1×
“Feature maps [8, 9, 10] and logits of a network [11, 12] are widely used as knowledge.”
BackgroundSelf-citationSmall-footprint keyword spotting using deep neural networks
Chen, G., Parada, C., & Heigold, G. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854370 · Cited by 545 · cited 1×
“However, with the increase of various voice assistants such as Siri, Hey Google, and Alexa on IoT devices, model compression has also become an important research topic in speech processing [13, 14, 15, 16, 17].”
Background
Cited by
A comprehensive review of model compression techniques in machine learning
Dantas, P. V., Sabino da Silva, W., Cordeiro, L. C., & Carvalho, C. B. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05747-w · Cited by 261
“For instance, while pruning and quantization effectively reduce model size and accelerate inference times, they may introduce artifacts or errors that could degrade model performance [19, 59, 116].”
A Comprehensive Survey of Deep Learning Approaches in Image Processing
Τρίγκα, Μ., & Dritsas, E. (2025) · Sensors · doi:10.3390/s25020531 · Cited by 207
A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
Katare, D., Perino, D., Nurmi, J., Warnier, M., Janssen, M., & Ding, A. Y. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3302474 · Cited by 158
“Current datasets vary in logging approaches which brings challenges to model or cross-data transformation which can also create a bias on the developed AI algorithm.”
BackgroundHardware-Aware DNN Compression via Diverse Pruning and Mixed-Precision Quantization
Balaskas, K., Καρατζάς, Α., Sad, C., Siozios, K., Anagnostopoulos, I., Zervakis, G., & Henkel, J. (2024) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2023.3346944 · Cited by 51
“PQK [32] is a distillation-based pruning framework which utilizes (previously unimportant) pruned and quantized weights in the distillation phase, instead of pretrained teacher models.”
MethodDeep Compression for Efficient and Accelerated Over-the-Air Federated Learning
Khan, F. M. A., Abou-Zeid, H., & Hassan, S. A. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3373460 · Cited by 41
“Pruning removes unimportant weights, which can then be used to create a teacher network for training a more efficient student network, eliminating the need for separate teacher training [25].”
BackgroundMethodInfluentialNaveen, S., & Kounte, M. R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20523-1 · Cited by 33
Model-as-a-Service (MaaS): A Survey
Gan, W., Wan, S., & Yu, P. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386351 · Cited by 79
“This includes techniques like model compression, quantization, and pruning [50], which reduce the model size and improve its efficiency without sacrificing performance.”
MethodQuantization Robust Pruning With Knowledge Distillation
Kim, J. (2023) · IEEE Access · doi:10.1109/access.2023.3257864 · Cited by 38
“in many ways to compress the model [2], [3], [4].”
MethodSelf-citationCooperative Inference of DNNs for Delay- and Memory-Constrained Wireless IoT Systems
Yun, S., Choi, W., & Kim, I. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3152359 · Cited by 27
“1Note that pruning and/or quantization can be applied on top of knowledge distillation as in [19] and [20] for our proposed scheme; but, it is beyond the scope of our work.”
Putrada, A. G., Abdurohman, M., Perdana, D., & Nuha, H. H. (2023) · IEEE Access · doi:10.1109/access.2023.3288425 · Cited by 26
“We compared three model compression methods to shrink our CIMA model for deployment in NodeMCU: pruning, quantization, and knowledge distillation [39].”
BackgroundMethodA Survey of Sustainability in Large Language Models: Applications, Economics, and Challenges
Singh, A., Patel, N. P., Ehtesham, A., Kumar, S., & Khoei, T. T. (2025) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc62904.2025.10903774 · Cited by 60
“– Distributed and Federated Learning : Implementing distributed and federated learning frameworks decentralizes computational loads and uses localized data processing, enhancing energy efficiency [60].”
InfluentialEfficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning
Malihi, L., & Heidemann, G. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7030154 · Cited by 20
Progressive multi-level distillation learning for pruning network
Wang, R., Wan, S., Wu, Z., Zhang, C., Li, Y., Xu, S., Zhang, L., Jin, X., Jiang, Z., & Rao, Y. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01036-0 · Cited by 23
“In addition,most of the previous studies have focused on the problem of how to improve the performance of unstructured pruning [13, 14], while there”
Memory optimization at Edge for Distributed Convolution Neural Network
Naveen, S., & Kounte, M. R. (2022) · Transactions on Emerging Telecommunications Technologies · doi:10.1002/ett.4648 · Cited by 23
Alahmari, S., & Alghamdi, I. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107849 · Cited by 21
Quantized Image Super-Resolution on Mobile Npus, Mobile AI 2025 Challenge: Report
Ignatov, A., Perevozchikov, G., Timofte, R., Zhang, Z., Gao, T., Yang, Y., Zhu, S., Wang, S., Yoon, K., Gankhuyag, G., et al. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00179 · Cited by 28
“…16-bit / 8-bit [7, 16, 56, 57, 62, 65, 70, 76, 77, 97] and low-bit [5, 8, 10, 54, 66, 84, 104] quantization, network pruning and compression [16, 25, 59–61, 67, 71, 88, 93, 103], NPU-specific adaptations, platform-aware neural architecture search [17, 18, 27, 69, 78, 86, 90, 91, 99, 100], etc .”
Elmessery, W. M., Shams, M. Y., El‐Hafeez, T. A., Szűcs, P., Eid, M. H., Alhumedi, M., Ahmed, A. F., & Elwakeel, A. E. (2025) · Scientific Reports · doi:10.1038/s41598-025-21677-0 · Cited by 15
“Kim et al 34 . proposed PQK, integrating pruning, quantization, and knowledge distillation in a two-phase framework specifically designed for edge deployment.”
Liang, Y., Xu, C., Mei, Y., Zou, H., Guo, J., Fan, X., Wang, T., & Huang, H. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3675866 · Cited by 19
“Similarly, PQK explicitly combines these three techniques into a unified pipeline, demonstrating e ff ective compression capabilities for image recognition tasks [75].”
Applications of Pruning Methods in Natural Language Processing
Touheed, M., Zubair, U., Sabir, D., Hassan, A., Butt, M. F. U., Riaz, F., Abdul, W., & Ayub, R. (2024) · IEEE Access · doi:10.1109/access.2024.3411776 · Cited by 17
“Because edge devices lack computing capacity for DNNs, Kim et al. [75] introduced a unique compression approach known as PQK. PQK uses pruning, quantization, and knowledge distillation (KD) to produce efficient models.”
BackgroundQuantization-Aware Training With Dynamic and Static Pruning
An, S., Shin, J., & Kim, J. (2025) · IEEE Access · doi:10.1109/access.2025.3556629 · Cited by 17
“Two predominant methods for quantization exist, namely Quantization Aware Training (QAT) [9], [10], [11], [12], [13], [14], [15] and Post-Training Quantization (PTQ) [16], [17], [18].”
Self-citationLearned Smartphone Isp on Mobile Gpus, Mobile Ai 2025 Challenge: Report
Ignatov, A., Perevozchikov, G., Timofte, R., Cheng, L., Liu, L., Cao, J., Sun, H., Pan, W., Wang, S., Yu, K., et al. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00181 · Cited by 22
“…mentioned limitations make the inference with standard deep learning architectures either inefficient or not even possible at all on mobile NPUs, therefore requiring various adaptations and low-level model design optimizations including 16-bit / 8-bit [5, 11, 50, 51, 59, 60, 65, 70, 72, 84] and…”
Matching the Ideal Pruning Method with Knowledge Distillation for Optimal Compression
Malihi, L., & Heidemann, G. (2024) · Applied System Innovation · doi:10.3390/asi7040056 · Cited by 18
A review of state-of-the-art techniques for large language model compression
Dantas, P. V., Cordeiro, L. C., & Waldir, S. S. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-02019-z · Cited by 32
“However, they also introduce intricate trade-offs [86].”
Efficient Convolutional Neural Networks With PWK Compression for Gas Pipelines Leak Detection
Meng, D., Ning, F., Hao, M., Xie, P., & Wei, J. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3457968 · Cited by 13
“In general, model compression can be divided into three types: pruning, quantification, and knowledge distillation (KD) [19].”
BackgroundHermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices
Han, X., Cai, Z., Zhang, Y., Fan, C., Liu, J., Ma, R., & Buyya, R. (2024) · IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD) · doi:10.1109/iccd63220.2024.00076 · Cited by 13
Markov-PQ: Joint Pruning-Quantization via Learnable Markov Chain
Li, Y., Zhang, X., Xie, W., Zhang, J., Fang, L., & Du, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3437182 · Cited by 9
“Inlightof the challenges posed by precise hyper-parameter tuning and coupled optimization problems, integrating pruning [23], [24], [25] to enhance the joint pruning-quantization framework.”
Qu, J., Luukkanen, S., & Wan, H. (2025) · Minerals Engineering · doi:10.1016/j.mineng.2025.109591 · Cited by 8
Xu, Y., Wang, J., Zhang, R., Zhao, C., Niyato, D. T., Kang, J., Xiong, Z., Qian, B., Zhou, H., Mao, S., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3641591 · Cited by 29
“To overcome these limitations, parameter-reduction strategies, such as expert pruning [267], quantization-aware training [268], and knowledge distillation [269], present promising solutions to selectively retain critical expert network weights, thereby reducing the computational overhead.”
Данилов, В., Витальевич, Л. В., Klyshnikov, K. Y., Stepanov, A. D., Bogdanov, L. A., Antonova, L. V., Krivkina, E. O., Kutikhin, A. G., & Ovcharenko, E. A. (2024) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2024.1411680 · Cited by 8
“…Ef fi ciency: For models like DeepLabV3 with high FLOPs, adopting techniques such as pruning, quantization, or knowledge distillation could reduce the computational load without signi fi cantly compromising accuracy (Li H. et al., 2016; Krishnamoorthi, 2018; Gou et al., 2021; Kim et al., 2021).”
MethodKishore, J., Jain, A., Koushika, K. K., Mishra, P. K., Karanwal, S., & Solanki, S. (2025) · Discover Computing · doi:10.1007/s10791-025-09637-8 · Cited by 8
“Kim et al. [48] provided strong experimental evidence supporting the superiority of KD over other compression methods, highlighting its effectiveness in preserving model performance even under aggressive compression settings.”
Feature-Domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution
Moon, H.-C., Kim, J., Jeong, J., & Kim, S. (2023) · IEEE Access · doi:10.1109/access.2023.3335934 · Cited by 7
“…of inheriting the knowledge of large teachers and improving performance without modifying the network structure which is already in place at the industrial site, 2) KD can be combined with pruning and network design methods by adding loss terms to achieve more performance improvement [1, 16].”
MethodSelf-citationTowards Energy Efficient DNN accelerator via Sparsified Gradual Knowledge Distillation
Karimzadeh, F., & Raychowdhury, A. (2022) · IFIP/IEEE International Conference on Very Large Scale Integration · doi:10.1109/vlsi-soc54400.2022.9939619 · Cited by 11
“PQK method pruned and quantizad the network at the same time in an interactive manner [14] while in our paper, we prune and quantize the network in the”
MethodRahman, M. H., Uddin, M. A., Ria, Z. F., & Rahman, R. M. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.058329 · Cited by 15
“The manuscript “Model Compression via Pruning, Quantization, and Knowledge Distillation,”authored by Kim et al. [4], presents the Pruning Quantization KD (PQK) approach, which amalgamates pruning, quantization, and KD for streamlined model compression [19].”
Lim, J., & Baek, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3314191 · Cited by 8
“for deep neural networks [5], [6], [7].”
Towards Low-Cost and Energy-Aware Inference for EdgeAI Services via Model Swapping
Trihinas, D., Michael, P., & Symeonides, M. (2024) · IEEE International Conference on Cloud Engineering · doi:10.1109/ic2e61754.2024.00026 · Cited by 9
“To minimize these effects, model compression techniques such as quantization (i.e., reduce precision of the model parameters) and pruning (i.e., reduce number of model parameters) have been proposed for resource-constraint AI services [18].”
Adversarial Machine Learning in IoT Security: A Comprehensive Survey
Huma, Z. E., Jan, S. U., Ahmad, J. E., Buchanan, W. J., & Pitropakis, N. (2025) · ACM Computing Surveys · doi:10.1145/3785665 · Cited by 15
“Model Robustness and Generalization: Resource Constraints Scalability:”
Devi, H., Kumar, P. R., Govindarajan, V., Kumar, S., Lohano, R., Hitesh, H., & Shiwlani, A. (2026) · IEEE Access · doi:10.1109/access.2026.3651446 · Cited by 5
“Futureresearch has to concentrate on creating scalable data collecting systems and examining on-device optimization strategies including quantization, model pruning, and knowledge distillation so that DL models can be deployed [61].”
Wang, Y., Zhang, S., Xu, J., Cheng, Z., & Du, G. (2025) · Symmetry · doi:10.3390/sym17071116 · Cited by 10
Convolutional neural network pruning based on misclassification cost
Ahmadluei, S., Faez, K., & Masoumi, B. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05487-7 · Cited by 5
“According to an innovative algorithm presented in [65], it is suggested that the teacher and student networks could be combined into a single network but separated by a mask on the filters.”
BackgroundYang, G.-P., Gu, Y., Macha, S., Tang, Q., & Liu, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447258 · Cited by 5
“The QAT strategies achieve model compression by reducing the bit size utilized to represent both weights and activations [15, 16, 17, 18, 19].”
BackgroundMethodDeep Learning Model Compression Techniques Performance on Edge Devices
Rachmanto, R. D., Nabhaan, A. N. L., & Setyanto, A. (2024) · Matrik Jurnal Manajemen Teknik Informatika dan Rekayasa Komputer · doi:10.30812/matrik.v23i3.3961 · Cited by 3
Anusha, S., & Manjith, R. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107930 · Cited by 1
Dual-Depth Unified Joint Optimization: Adaptive Curvature-Based Compression
Li, Y., Zhang, X., Xie, W., Chen, X., Li, D., Ye, H., & Fang, L. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3579007 · Cited by 5
“The first metric is based on the PQK method [50], and the others are derived from the Attention Transfer framework [51].”
Overview of Memory-Efficient Architectures for Deep Learning in Real-Time Systems
Demir, B., Domazet, E., & Mechkaroska, D. (2025) · International Conference on Electronics, Engineering Physics and Earth Science (EEPES) · doi:10.3390/engproc2025104077 · Cited by 5
Muhamad, N., Li, X., Imtiaz, S., et al. (2026) · IET Renewable Power Generation · doi:10.1049/rpg2.70200 · Cited by 2
Model Compression via Position-Based Scaled Gradient
Kim, J., Yoo, K., & Kwak, N. (2022) · IEEE Access · doi:10.1109/access.2022.3231455 · Cited by 2
“Model pruning methods [23], [24], [25], [26], [27], [28], [29], [30] try to prune weights or filters in the model considered unimportant units by proposed unique criterion [31], [32].”
MethodSelf-citationReusing ML Models in Dynamic Data Environments: Data Similarity-Based Approach for Efficient MLOps
Peixoto, E., Torres, D., Carneiro, D. R., Silva, B., & Marques, R. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9020047 · Cited by 6
“Frugal AI techniques include computational optimizations such as model pruning, quantization, distillation, and leveraging pre-trained models through transfer learning [9].”
Rgb Photo Enhancement on Mobile Gpus, Mobile Ai 2025 Challenge: Report
Ignatov, A., Perevozchikov, G., Timofte, R., Pan, W., Wang, S., Zhang, D., Zhao, R., Li, X., Ju, S., Zhang, D., et al. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00180 · Cited by 21
“…16-bit / 8-bit [4, 8, 44, 45, 52, 53, 61, 68, 69, 82] and low-bit [3, 5, 7, 43, 55, 72, 86] quantization, network pruning and compression [8, 14, 48, 49, 51, 56, 62, 76, 81, 85], NPU-specific adaptations, platform-aware neural architecture search [9, 10, 15, 60, 70, 74, 79, 80, 83, 84], etc .”
Enhancing Edge-Based Mango Pest Classification Through Model Optimization
Tran, L. D., & Tran, M. (2023) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf60135.2023.10471857 · Cited by 2
“With limited computational resources, [17] introduces PQK model compression consisting of pruning, quantization, and knowledge distillation (KD) [6] processes.”
BackgroundArch-Net: Model conversion and quantization for architecture agnostic model deployment
Fang, S., Xu, W., Feng, Z., Yuan, S. M., Wang, Y., Yang, Y., Ding, W., & Zhou, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107384 · Cited by 2
Trihinas, D., Symeonides, M., Cleju, N., Pallis, G., & Dikaiakos, M. (2025) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1145/3773274.3774270 · Cited by 3
“Focusing on more cost-effective solutions at the DL model level, others suggest applying pruning or quantization (e.g., reduce number and precision of model parameters) [10].”
Performance optimizations on U-Net speech enhancement models
Chee, J., Braun, S., Gopal, V., & Cutler, R. (2022) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp55362.2022.9949392 · Cited by 1
“[2] use a combination of unstructured pruning, quantization, and knowledge distillation to compress keyword spotting models.”
MethodIntelligent Computer Vision Systems in the Processing of Baggage and Hand Luggage X-ray Images
Andriyanov, N. A. (2024) · Learning and analytics in intelligent systems · doi:10.1007/978-3-031-62316-5_12 · Cited by 1
PCA-based compression of deep LSTMs: a robust framework for uncertain industrial systems
Yahyaoui, Z., Hajji, M., & Bouazzi, Y. (2025) · Computing · doi:10.1007/s00607-025-01565-1 · Cited by 1
“This pro - cess involves several techniques [ 11 ], including pruning [ 12, 13 ], quantization [ 14, 15 ], and knowledge distillation [ 16 ].”
Performance optimizations on deep noise suppression modelsPreprint
Chee, J., Braun, S., Gopal, V., & Cutler, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04378
“There has been much work in compressing deep learning methods so that they can efficiently operate within the realtime and hardware constraints of many audio enhancement applications [1, 2, 3, 4].”
BackgroundMethodCloud-Edge Intelligent Collaborative Computing Model Based on Transfer Learning in IoT
Long, Y., & Li, Z. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-5968-6_28
Language Technology-based Political Microtargeting Services
García-Díaz, J. A., Marín-Pérez, M. J., Mármol, G. A., Almela, Á., Miñarro-Giménez, J. A., & García‐Sánchez, F. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-75702-0_6
Thaker, P., & Asif, A. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-3863-5_8
Dey, R., Murugesan, S., & Mazumde, R. (2025) · doi:10.1109/wconf64849.2025.11233758
“Kim et al. [4] suggests employing the following methods for more energy-efficient training: • Model Pruning (Cut the fat): The goal is to gradually remove ”unimportant” weights or even layers (e.g., low magnitudes) during training to reduce redundancy.”
InfluentialPang, Y., Zhou, G., Meng, X., Wang, H., Liu, H., Pan, X., & Chen, B. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3635209 · Cited by 1
Co-Optimization and Deployment of Lightweight Convolutional Neural Networks on FPGAs
Huang, J. (2025) · International Conference on Electronic Information Engineering and Computer Technology (EIECT) · doi:10.1109/eiect68017.2025.11331770
“To overcome this, lightweight strategies such as quantization, pruning, and knowledge distillation[1] have been developed to reduce model size while maintaining accuracy.”
A Multi Stage Framework for Efficient Deep Neural Network Compression on Edge Devices
Dubey, S. K., & Ahmad, N. (2025) · International Conference on Intelligent Systems and Embedded Design (ISED) · doi:10.1109/ised67359.2025.11405082
Uncertainty-Guided Bandwidth-Adaptive Model Compression for Industrial Pore Detection
Roy, P., Lee, K., Andani, M. T., et al. (2026) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/icnc68183.2026.11416946
“PQK is a pruning-quantization-knowledge distillation based approach for edge devices aware compression [11].”
Vers une Intelligence Artificielle Numérique Responsable
Ciocan, A. V. (2024) · theses.fr (ABES) · doi:10.70675/56c5e5d9z8984z4a72zba0bzf0e2f7041cc7 · Cited by 1
Hybrid Deep Learning and Statistical Forecasting for Short Term Electric Load
Brodjian, S., Liao, J., Goswami, A., et al. (2026) · Proceedings of IEEE Southeastcon · doi:10.1109/southeastcon63549.2026.11476233
“On the efficiency front, advancements in model pruning and knowledge distillation have enabled the development of lightweight models that do not compromise on accuracy while being more amenable to deployment on less powerful hardware [14] [15].”
Zhou, Y.-E., Ma, L., Wang, X., Li, Q., Chiasserini, C. F., & Han, G. (2026) · IEEE Transactions on Networking · doi:10.1109/ton.2026.3689748
“Its aim is to convert full-precision models to low-bit lightweight ones with no / minor performance degradation [24], [31], [36], [44], where, e.g., for 32-bit full precision, the 8-bit quantization refers to 75% compression ratio of model, and so forth.”
Energy-Efficient Transformer Architectures for on-device Language Understanding
Jayaraju, P., Pawar, M., Haldorai, A., Salati, A., Sivaramakrishnan, A., & Kumar, G. R. (2026) · International Conference on Recent Trends in Computer Science and Technology (ICRTCST) · doi:10.1109/icrtcst68392.2026.11545410
“Various methods like model pruning, quantization, knowledge distillation, and architecture optimization have been experimented with to tackle these issues [7].”
Anti, N. Y. D., Frimpong, S., & Raza, M. A. (2026) · Sensors · doi:10.3390/s26134258 · Cited by 1
“Surveys high-light these as the core compression tools for edge deployment, often used jointly [59,60].”
Deep Learning Methodologies for Oil Spill Prevention: A Lifecycle-Based Sustainability Review
Malang, K. A., Ezenkwu, C. P., & Moreno-García, C. F. (2026) · Petroleum Research · doi:10.1016/j.ptlrs.2026.07.001
HPQ: A Hybrid Framework for Joint Pruning and Quantization of Self-Supervised Speech Models
Peng, J., Zhang, L., Han, J., Plchot, O., Wang, S., & Černocký, J. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3712284
Optimizing Graph Neural Networks for Money Laundering Detection via Model Compression
Kinasih, D. P., Doloksaribu, M. J., Michael, et al. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11620900
Aliyu, I., Hassan, I. H., Mohammed, M. T., Iorzua, J. T., Abdullahi, M., Yusuf, A., & Kana, A. D. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134614
Visual-Based Flower Counting: Techniques and Applications
Zang, S., Shu, L., Han, R., Yang, X. K., Cielniak, G., & Zhang, F. (2026) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2026.126050
CHURI: Contour-Gated, Resource-Intelligent Retail Tracking System
Waghmare, R. (2026) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw68408.2026.00170
Zhang, H., Hu, Y., Yuan, X., & Schmeink, A. (2026) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2026.3718740
“Lastly, the Kullback-Leibler (KL) divergence measures the distributional fidelity of a quantized feature channel relative to its full-precision distribution [39].”
Bourouis, A., Nasri, A., Zaidi, S., et al. (2026) · Vehicles · doi:10.3390/vehicles8090215
Hermes \({}^{+}\) : Adaptive Memory-Efficient Pipeline Inference for Large Models on Edge Devices
Cai, Z., Han, X., Jiang, N., Zhang, B., Qiang, Y., Wu, T., Liu, Y., & Ma, R. (2026) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3845607
A Review of Lightweight Algorithm‐Driven Intelligent Sensing Technologies for Extreme Environments
Wang, Y., Li, S., Xiao, Z., Tian, L., Yang, C., Song, J., Zheng, L., Zhang, L., & Lawrie, C. H. (2026) · Journal of Sensors · doi:10.1155/js/3919320
Hasan, M. M., Rahman, A., Jannat, U. M., et al. (2026) · IEEE Region 10 Symposium (TENSYMP) · doi:10.1109/tensymp68909.2026.11704062
“The deployment of high capacity models on memory and power constrained embedded or vehicular platforms remains critically challenging, motivating research into efficient architectural frameworks and compression aware optimization [7].”
赵林昌, Deng, J., Hu, G., Zhang, M., Xu, Y., & Zhen, L. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.113224
Kumar, S. Y. (2026) · Proceedings of the Institution of Mechanical Engineers Part B Journal of Engineering Manufacture · doi:10.1177/09544054261492096
Metadata
- Authors
- Kim, J., Chang, S., & Kwak, N.
- Affiliations
- Seoul National University (KR)
- Qualcomm (United Kingdom) (GB)
- Venue
- Interspeech 2021
- Publisher
- ISCA
- Published
- 2021-08-30
- Pages
- 4568-4572
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3174517946
- Semantic Scholar
- fd504f20b3ce6cb4e09633941cefa388311d30ce
- Cited by
- 85
- References
- 20
- Citations by year
- 2026: 21 · 2025: 29 · 2024: 17 · 2023: 11 · 2022: 6 · 2021: 1
- FWCI
- 2.86
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- model compression, neural network pruning, quantization-aware training, knowledge distillation, edge devices, lightweight neural networks, teacher-student models, keyword spotting, image recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.