CoChat

Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning

Open access
Malihi, Leila, Heidemann, Gunther · 2023 · Big Data and Cognitive Computing

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 14 of the 19 sources its claims cite were checked for support.

Abstract

Efficient model deployment is a key focus in deep learning. This has led to the exploration of methods such as knowledge distillation and network pruning to compress models and increase their performance. In this study, we investigate the potential synergy between knowledge distillation and network pruning to achieve optimal model efficiency and improved generalization. We introduce an innovative framework for model compression that combines knowledge distillation, pruning, and fine-tuning to achieve enhanced compression while providing control over the degree of compactness. Our research is conducted on popular datasets, CIFAR-10 and CIFAR-100, employing diverse model architectures, including ResNet, DenseNet, and EfficientNet. We could calibrate the amount of compression achieved. This allows us to produce models with different degrees of compression while still being just as accurate, or even better. Notably, we demonstrate its efficacy by producing two compressed variants of ResNet 101: ResNet 50 and ResNet 18. Our results reveal intriguing findings. In most cases, the pruned and distilled student models exhibit comparable or superior accuracy to the distilled student models while utilizing significantly fewer parameters.

Overview

CoChat Summary

TL;DR

The paper proposes a sequential compression pipeline that applies SimKD knowledge distillation first, then L1 unstructured pruning, then fine-tuning only the last fully connected layer. Across CIFAR-10 and CIFAR-100 with 11 teacher-student model combinations, it reports substantial parameter reduction while often matching or exceeding teacher accuracy.

Key findings

  • SimKD consistently outperformed the compared distillation methods on CIFAR-100 and CIFAR-10, and its distilled students were selected for pruning.
  • Pruned student models often achieved comparable or higher accuracy than teachers with far fewer parameters; the paper reports more than 70% compression of the teacher model across 11 model combinations, with some cases over 90%.
  • For ResNet101→ResNet50 on CIFAR-100, the pruned student reached 58.92% accuracy with 0.5 × 10^7 parameters versus the teacher’s 54.07% with 4.45 × 10^7 parameters, an 88.76% parameter reduction. On CIFAR-10, the same pair reached 83.51% with 0.6 × 10^7 parameters versus the teacher’s 81.1% with 4.45 × 10^7 parameters, an 86.51% reduction.
  • The paper reports two compressed ResNet101 variants: one pruned from ResNet50 with a 7.41× compression rate and 83.5% accuracy, and one pruned from ResNet18 with a 13.08× compression rate and 80.78% accuracy, about 1% lower than its original version.

Methods

  • Pipeline: knowledge distillation using SimKD with teacher classifier reuse and feature alignment, followed by L1 unstructured pruning and fine-tuning of the last fully connected layer.
  • Data and models: CIFAR-10 and CIFAR-100, with ResNet, DenseNet, EfficientNet, and MobileNet architectures across 11 teacher-student combinations; models were pretrained on ImageNet.
  • Training and analysis: SGD with 0.9 Nesterov momentum, 50 epochs, learning rate 0.001, batch size 64, weight decay 5 × 10^-4, temperature T = 4, PyTorch 2.0.0 on a Quadro RTX 8000. Pruning reduced parameters from 0% to 90% in 10% increments, repeated 10 times per teacher-student pair and averaged; accuracy and remaining parameters were compared, including a “breakpoint” threshold.

Limitations

  • Stated limitations: The paper does not include a dedicated limitations section; it instead lists future work such as combining KD with structured or channel pruning, optimizing fine-tuning, and evaluating on wider and more challenging datasets.
  • Evident from the text: Evaluation is limited to CIFAR-10 and CIFAR-100 after ImageNet pretraining, and fine-tuning is applied only to the last fully connected layer rather than the whole network.
  • Evident from the text: The pruning experiments are conducted on SimKD-distilled student models, so the reported pruning results are tied to that specific distillation choice.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/bdcc7030154

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

5 Valid · 1 Not valid · 4 Not checked · 42 Own results and interpretations

23 more claims

Create a free account to see all 52 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 3
  • It may state two separate claims as one. 22
  • Borderline: one of the tests for a substantial claim is unclear. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 21 references

Research works printed but not linked: 1

  1. LeCun, Y., Denker, J., and Solla, S. (1989). Proceedings of the Advances in Neural Information Processing Systems, Morgan-Kaufmann.

Cited by

Metadata

Authors
Malihi, L., & Heidemann, G.
Affiliations
Osnabrück University (DE)
Corresponding author
Malihi, Leila
Venue
Big Data and Cognitive Computing
Publisher
MDPI AG
Published
2023-09-19
Volume
7
Issue
3
Pages
154
ISSN
2504-2289 (ISSN-L)
OpenAlex
W4386895277
Cited by
20
References
14
Citations by year
2026: 6 · 2025: 10 · 2024: 4
FWCI
1.82
Type
journal article
Language
en
Open access
Locations
  • Big Data and Cognitive Computing (publishedVersion, cc-by)
  • osnaDocs (Osnabrück University) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • osnaDocs (Osnabrück University) (cc-by)
Funders
Osnabrück University
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
model compression, knowledge distillation, network pruning, fine-tuning, ResNet, EfficientNet, DenseNet, CIFAR-10, CIFAR-100, parameter reduction, generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.