The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index; 8 of the 13 sources its claims cite were checked for support.
PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation
Open accessAbstract
As edge devices become prevalent, deploying Deep Neural Networks (DNN) on edge devices has become a critical issue. However, DNN requires a high computational resource which is rarely available for edge devices. To handle this, we propose a novel model compression method for the devices with limited computational resources, called PQK consisting of pruning, quantization, and knowledge distillation (KD) processes. Unlike traditional pruning and KD, PQK makes use of unimportant weights pruned in the pruning process to make a teacher network for training a better student network without pre-training the teacher model. PQK has two phases. Phase 1 exploits iterative pruning and quantization-aware training to make a lightweight and power-efficient model. In phase 2, we make a teacher network by adding unimportant weights unused in phase 1 to a pruned network. By using this teacher network, we train the pruned network as a student network. In doing so, we do not need a pre-trained teacher network for the KD framework because the teacher and the student networks coexist within the same network. We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.
Overview
CoChat Summary
TL;DR
PQK is a model-compression framework that combines iterative pruning, quantization-aware training, and knowledge distillation by reusing pruned unimportant weights to form a teacher network without pre-training a separate teacher. It is evaluated on keyword spotting and image recognition, where phase 2 KD improves pruned/quantized student accuracy over phase 1.
Key findings
- On Google Speech Commands with ResNet-8-narrow at target pruning ratio \(p_t=0.9\), phase 2 improves the 4-bit pruned net from 74.1% to 83.1% and the 8-bit pruned net from 81.7% to 86.4%, gains of 9.0% and 4.7% respectively.
- On ResNet-8 at \(p_t=0.9\), phase 2 improves 8-bit pruned net accuracy from 92.6% to 94.0% and 4-bit from 91.7% to 93.6%; phase2-F reaches 94.4% at 8-bit versus 94.3% vanilla.
- On CIFAR100 with ResNet-32, phase2-F with an 8-bit pruned net reaches 71.1% versus 69.7% vanilla, outperforming vanilla by 1.4%; phase2-P reaches 69.8%.
- Ablation on ResNet-8-narrow 4-bit \(p_t=0.9\): phase2-P reaches 83.1%, outperforming the best finetuning baseline (lr=0.1, 79.2%) by 3.9%.
Methods
- PQK has two phases: phase 1 uses iterative magnitude-based unstructured pruning with a gradually increasing pruning ratio from \(p_i=0\) to target \(p_t\), updating the mask every 32 iterations, plus uniform symmetric per-layer quantization-aware training with a learnable step-size and straight-through estimator; phase 2 builds a full-net teacher by adding unimportant weights to the pruned student and trains both with KD using KL and cross-entropy losses, with a warm-up stage (\(\alpha=1,\beta=0\)) before setting \(T=2,\alpha=0.5,\beta=0.5\).
- Experiments use Google Speech Commands Dataset v1 with ResNet-8 and ResNet-8-narrow, and CIFAR100 with ResNet-32; target pruning ratio is 0.9 for KWS, with additional ResNet-8 runs at \(p_t\in\{0.9,0.7,0.5\}\), and quantization is 8-bit and 4-bit against a 32-bit baseline.
- Phase 2 adds 9 epochs for KWS and 60 epochs for CIFAR100; evaluation compares vanilla, phase1-P, phase2-P, and phase2-F test accuracy, plus an ablation against finetuning with the same training budget.
Limitations
- The paper states no limitations.
- Evident from the text: hyperparameters were not grid-searched but chosen from related works, and evaluation is limited to two tasks/datasets and specific ResNet models.
- Evident from the text: phase 2 requires additional training epochs, two forward passes, and different batchnorm parameters for pruned and full nets; the full net remains 32-bit while sharing the quantized pruned net, and combining 32-bit with 4-bit trained models is noted as more unstable than with 8-bit.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 3 Not valid · 0 Not checked · 28 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Concerning bitwidth and accuracy, 8-bit consistently outperforms 4-bit because of its high representative power from more bitwidth.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, there is a large performance gap between phase 1 and 2 in the pruned model of ResNet-8-narrow compared to that of ResNet-8.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In ResNet-8 with 4-bit, although pruned net performs well, accuracies of phase2-F are lower than those of phase2-P.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table ( p t = 0 . 9 ), the performance of phase1P decreases compared to vanilla.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
15 more claims
Create a free account to see all 35 claims and to verify the 4 references still open.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 8
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1. Experimental Setup · p. 3
“We set the warm up iteration ( s ) as 1500.”
§ Algorithm 1 PQK · p. 2
“13: for Epoch = 1 ,..., P 2 do 14: if s < Epoch then 15: set α, β // KD training after warm up stage”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“Image Recognition We use CIFAR100 Dataset [23] and choose ResNet-32 [24] for the baseline.”
MethodLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Feature maps [8, 9, 10] and logits of a network [11, 12] are widely used as knowledge.”
BackgroundEstimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910 · cited 4×
“Dequantization step just brings the quantized value back to the original range by multiplying the step-size: w̄ = ŵ × Sw. (3) These quantization and dequantization processes are nondifferentiable, so we utilize a straight-through estimator (STE) [19] for backpropagation.”
MethodInfluentialHubara, I., Soudry, D., & Yaniv, R. E. (2016) · arXiv · doi:10.48550/arxiv.1602.02505 · Cited by 1,667
Learning both Weights and Connections for Efficient Neural Networks
Han, S., Pool, J., Tran, J., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1506.02626 · Cited by 1,481
Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
He, Y., Liu, P., Wang, Z., Hu, Z., & Yang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00447 · Cited by 1,217 · cited 1×
“(1) Pruning method prunes the unimportant weights or channels based on different criteria [1, 2, 3, 4].”
MethodQuantizing deep convolutional networks for efficient inference: A whitepaper
Krishnamoorthi, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08342 · Cited by 753
A Comprehensive Overhaul of Feature Distillation
Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., & Choi, J. Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00201 · Cited by 706 · cited 1×
“Feature maps [8, 9, 10] and logits of a network [11, 12] are widely used as knowledge.”
BackgroundSelf-citationSmall-footprint keyword spotting using deep neural networks
Chen, G., Parada, C., & Heigold, G. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854370 · Cited by 545 · cited 1×
“However, with the increase of various voice assistants such as Siri, Hey Google, and Alexa on IoT devices, model compression has also become an important research topic in speech processing [13, 14, 15, 16, 17].”
Background
Cited by
A comprehensive review of model compression techniques in machine learning
Dantas, P. V., Sabino da Silva, W., Cordeiro, L. C., & Carvalho, C. B. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05747-w · Cited by 261
“For instance, while pruning and quantization effectively reduce model size and accelerate inference times, they may introduce artifacts or errors that could degrade model performance [19, 59, 116].”
A Comprehensive Survey of Deep Learning Approaches in Image Processing
Τρίγκα, Μ., & Dritsas, E. (2025) · Sensors · doi:10.3390/s25020531 · Cited by 206
A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
Katare, D., Perino, D., Nurmi, J., Warnier, M., Janssen, M., & Ding, A. Y. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3302474 · Cited by 153
“Current datasets vary in logging approaches which brings challenges to model or cross-data transformation which can also create a bias on the developed AI algorithm.”
BackgroundHardware-Aware DNN Compression via Diverse Pruning and Mixed-Precision Quantization
Balaskas, K., Καρατζάς, Α., Sad, C., Siozios, K., Anagnostopoulos, I., Zervakis, G., & Henkel, J. (2024) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2023.3346944 · Cited by 45
“PQK [32] is a distillation-based pruning framework which utilizes (previously unimportant) pruned and quantized weights in the distillation phase, instead of pretrained teacher models.”
MethodDeep Compression for Efficient and Accelerated Over-the-Air Federated Learning
Khan, F. M. A., Abou-Zeid, H., & Hassan, S. A. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3373460 · Cited by 40
“Pruning removes unimportant weights, which can then be used to create a teacher network for training a more efficient student network, eliminating the need for separate teacher training [25].”
BackgroundMethodInfluential
Metadata
- Authors
- Kim, J., Chang, S., & Kwak, N.
- Affiliations
- Seoul National University (KR)
- Qualcomm (United Kingdom) (GB)
- Venue
- Interspeech 2021
- Publisher
- ISCA
- Published
- 2021-08-30
- Pages
- 4568-4572
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3174517946
- Semantic Scholar
- fd504f20b3ce6cb4e09633941cefa388311d30ce
- Cited by
- 85
- References
- 20
- Citations by year
- 2026: 21 · 2025: 29 · 2024: 17 · 2023: 11 · 2022: 6 · 2021: 1
- FWCI
- 2.86
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- model compression, neural network pruning, quantization-aware training, knowledge distillation, edge devices, lightweight neural networks, teacher-student models, keyword spotting, image recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation. https://app.cochat.ai/papers/10.21437/interspeech.2021-248
- The paper
- Kim, J., Chang, S., & Kwak, N. (2021). PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation. Interspeech 2021. https://doi.org/10.21437/interspeech.2021-248
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.