CoChat

PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation

Open access
Kim, Jangho, Chang, Simyung, Kwak, Nojun · 2021 · Interspeech 2021

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index; 8 of the 13 sources its claims cite were checked for support.

Abstract

As edge devices become prevalent, deploying Deep Neural Networks (DNN) on edge devices has become a critical issue. However, DNN requires a high computational resource which is rarely available for edge devices. To handle this, we propose a novel model compression method for the devices with limited computational resources, called PQK consisting of pruning, quantization, and knowledge distillation (KD) processes. Unlike traditional pruning and KD, PQK makes use of unimportant weights pruned in the pruning process to make a teacher network for training a better student network without pre-training the teacher model. PQK has two phases. Phase 1 exploits iterative pruning and quantization-aware training to make a lightweight and power-efficient model. In phase 2, we make a teacher network by adding unimportant weights unused in phase 1 to a pruned network. By using this teacher network, we train the pruned network as a student network. In doing so, we do not need a pre-trained teacher network for the KD framework because the teacher and the student networks coexist within the same network. We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.

Overview

CoChat Summary

TL;DR

PQK is a model-compression framework that combines iterative pruning, quantization-aware training, and knowledge distillation by reusing pruned unimportant weights to form a teacher network without pre-training a separate teacher. It is evaluated on keyword spotting and image recognition, where phase 2 KD improves pruned/quantized student accuracy over phase 1.

Key findings

  • On Google Speech Commands with ResNet-8-narrow at target pruning ratio \(p_t=0.9\), phase 2 improves the 4-bit pruned net from 74.1% to 83.1% and the 8-bit pruned net from 81.7% to 86.4%, gains of 9.0% and 4.7% respectively.
  • On ResNet-8 at \(p_t=0.9\), phase 2 improves 8-bit pruned net accuracy from 92.6% to 94.0% and 4-bit from 91.7% to 93.6%; phase2-F reaches 94.4% at 8-bit versus 94.3% vanilla.
  • On CIFAR100 with ResNet-32, phase2-F with an 8-bit pruned net reaches 71.1% versus 69.7% vanilla, outperforming vanilla by 1.4%; phase2-P reaches 69.8%.
  • Ablation on ResNet-8-narrow 4-bit \(p_t=0.9\): phase2-P reaches 83.1%, outperforming the best finetuning baseline (lr=0.1, 79.2%) by 3.9%.

Methods

  • PQK has two phases: phase 1 uses iterative magnitude-based unstructured pruning with a gradually increasing pruning ratio from \(p_i=0\) to target \(p_t\), updating the mask every 32 iterations, plus uniform symmetric per-layer quantization-aware training with a learnable step-size and straight-through estimator; phase 2 builds a full-net teacher by adding unimportant weights to the pruned student and trains both with KD using KL and cross-entropy losses, with a warm-up stage (\(\alpha=1,\beta=0\)) before setting \(T=2,\alpha=0.5,\beta=0.5\).
  • Experiments use Google Speech Commands Dataset v1 with ResNet-8 and ResNet-8-narrow, and CIFAR100 with ResNet-32; target pruning ratio is 0.9 for KWS, with additional ResNet-8 runs at \(p_t\in\{0.9,0.7,0.5\}\), and quantization is 8-bit and 4-bit against a 32-bit baseline.
  • Phase 2 adds 9 epochs for KWS and 60 epochs for CIFAR100; evaluation compares vanilla, phase1-P, phase2-P, and phase2-F test accuracy, plus an ablation against finetuning with the same training budget.

Limitations

  • The paper states no limitations.
  • Evident from the text: hyperparameters were not grid-searched but chosen from related works, and evaluation is limited to two tasks/datasets and specific ResNet models.
  • Evident from the text: phase 2 requires additional training epochs, two forward passes, and different batchnorm parameters for pruned and full nets; the full net remains 32-bit while sharing the quantized pruned net, and combining 32-bit with 4-bit trained models is noted as more unstable than with 8-bit.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.21437/interspeech.2021-248

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 3 Not valid · 0 Not checked · 28 Own results and interpretations

15 more claims

Create a free account to see all 35 claims and to verify the 4 references still open.

See all claims — free account
  • Probably not a substantial claim. 11
  • It may state two separate claims as one. 8

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Cited by

See all 85

Metadata

Authors
Kim, J., Chang, S., & Kwak, N.
Affiliations
  • Seoul National University (KR)
  • Qualcomm (United Kingdom) (GB)
Venue
Interspeech 2021
Publisher
ISCA
Published
2021-08-30
Pages
4568-4572
ISSN
2308-457X (ISSN-L)
OpenAlex
W3174517946
Cited by
85
References
20
Citations by year
2026: 21 · 2025: 29 · 2024: 17 · 2023: 11 · 2022: 6 · 2021: 1
FWCI
2.86
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Interspeech (USB) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Seoul National University Open Repository (Seoul National University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
model compression, neural network pruning, quantization-aware training, knowledge distillation, edge devices, lightweight neural networks, teacher-student models, keyword spotting, image recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.