The DOI matches its Crossref record; 39 of the 43 references it cites were found in a registry or an index.
Knowledge from the original network: restore a better pruned network with knowledge distillation
Open accessAbstract
To deploy deep neural networks to edge devices with limited computation and storage costs, model compression is necessary for the application of deep learning. Pruning, as a traditional way of model compression, seeks to reduce the parameters of model weights. However, when a deep neural network is pruned, the accuracy of the network will significantly decrease. The traditional way to decrease the accuracy loss is fine-tuning. When over many parameters are pruned, the pruned network’s capacity is reduced heavily and cannot recover to high accuracy. In this paper, we apply the knowledge distillation strategy to abate the accuracy loss of pruned models. The original network of the pruned network was used as the teacher network, aiming to transfer the dark knowledge from the original network to the pruned sub-network. We have applied three mainstream knowledge distillation methods: response-based knowledge, feature-based knowledge, and relation-based knowledge (Gou et al. in Knowledge distillation: a survey. arXiv:200605525, 2020), and compare the result to the traditional fine-tuning method with grand-truth labels. Experiments have been done on the CIFAR100 dataset with several deep convolution neural network. Results show that the pruned network recovered by knowledge distillation with its original network performs better accuracy than it recovered by fine-tuning with sample labels. It has also been validated in this paper that the original network as the teacher performs better than differently structured networks with same accuracy as the teacher.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our approach provides a clear advantage by enabling a greater compression of the teacher model compared to the previous methods that relied on pruning [10-15], knowledge distillation (KD) [1,3,5,7,8], and the combination of both [16-20].”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
Li, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931
(1989) · Cited by 3,214
Pruning Filters for Efficient ConvNets
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016) · arXiv · doi:10.48550/arxiv.1608.08710 · Cited by 2,928
Research works printed but not linked: 1
- Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. Adv Neural Inf Process Syst 2015:1135–1143
Cited by
Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels
Wang, Y., Wang, H., Shen, Y., Fei, J., Li, W., Jin, G., Wu, L., Zhao, R., & Le, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00421 · Cited by 504
“In semi-supervised learning tasks, discarding unreliable pseudo-labels or reducing their weights is widely used to prevent degradation of model’s performance [5, 6, 38, 42, 44, 51].”
BackgroundSelf-citationA Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Cheng, H., Zhang, M., & Shi, J. Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447085 · Cited by 497
Towards edge computing in intelligent manufacturing: Past, present and future
Nain, G., Pattanaik, K. K., & Sharma, G. K. (2022) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2022.01.010 · Cited by 283
Zhong, H., Yu, S. S., Trinh, H. M., Lv, Y., Yuan, R. B., & Wang, Y. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3307425 · Cited by 26
“Chen et al. [18] postulated and showed that knowledge distillation could be integrated with network pruning.”
BackgroundLightweight citrus leaf disease detection model based on ARMS and cross-domain dynamic attention
Mo, H., & Wei, L. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102133 · Cited by 30
“In order to deploy lightweight models on mobile devices, the models not only need to maintain high accuracy, but also need to meet size and resource constraints (Chen et al., 2021).”
Metadata
- Authors
- Chen, L., Chen, Y., Xi, J., & Le, X.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Corresponding author
- Le, Xinyi
- Venue
- Complex & Intelligent Systems
- Published
- 2021-01-10
- Volume
- 8
- Issue
- 2
- Pages
- 709-718
- ISSN
- 2198-6053 (ISSN-L)
- 2199-4536
- OpenAlex
- W3119460773
- Cited by
- 48
- References
- 43
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 14 · 2023: 7 · 2022: 8 · 2021: 6
- FWCI
- 3.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Complex & Intelligent Systems (publishedVersion, cc-by)
- Funders
- Open Fund of Shenzhen Institue of Artificial Intelligence and Robotics for Societ (AC01202005016)
- Aeronautical Science Foundation of China (2019ZE057001)
- Shanghai Rising-Star Program (20QC1401100)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- knowledge distillation, neural network pruning, model compression, teacher networks, fine-tuning, edge deployment, convolutional neural networks, CIFAR-100
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Knowledge from the original network: restore a better pruned network with knowledge distillation. https://app.cochat.ai/papers/10.1007/s40747-020-00248-y
- The paper
- Chen, L., Chen, Y., Xi, J., & Le, X. (2022). Knowledge from the original network: restore a better pruned network with knowledge distillation. Complex & Intelligent Systems. https://doi.org/10.1007/s40747-020-00248-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.