CoChat
Sign Up
Cited by 335Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

The DOI matches its Crossref record, with a small difference in title or year; 31 of the 31 references it cites were found in a registry or an index.

Cross-Layer Distillation with Semantic Calibration

Open access
Chen, Defang, Mei, Jian-Ping, Zhang, Yuan, Wang, Can, Wang, Zhe, Feng, Yan, Chen, Chun·2021·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v35i8.16865
Open this paper in CoChat — free account

Abstract

Recently proposed knowledge distillation approaches based on feature-map transfer validate that intermediate layers of a teacher model can serve as effective targets for training a student model to obtain better generalization ability. Existing studies mainly focus on particular representation forms for knowledge transfer between manually specified pairs of teacher-student intermediate layers. However, semantics of intermediate layers may vary in different networks and manual association of layers might lead to negative regularization caused by semantic mismatch between certain teacher-student layer pairs. To address this problem, we propose Semantic Calibration for Cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple layers rather than a single fixed intermediate layer from the teacher model for appropriate cross-layer supervision in training. Consistent improvements over state-of-the-art approaches are observed in extensive experiments with various network architectures for teacher and student models, demonstrating the effectiveness and flexibility of the proposed attention based soft layer association mechanism for cross-layer distillation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v35i8.16865

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 29 references

Research works printed but not linked: 34

  1. [2][2] S. Han, H. Mao, and W. J. Dally, 'Deep compression: Compressing deep neural network with pruning, trained quantization and huffmancoding,' in International Conference on Learning Representations , 2016. -
  2. [3][3] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, 'Understanding deep learning requires rethinking generalization,' in International Conference on Learning Representations , 2017. -
  3. [4][4] K. He, X. Zhang, S. Ren, and J. Sun, 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,' in International Conference on Computer Vision , 2015, pp. 1026-1034. -
  4. [5][5] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, 'Dropout: a simple way to prevent neural networks from overfitting,' Journal of Machine Learning Researh , vol. 15, no. 1, pp. 1929-1958, 2014. -
  5. [6][6] S. Ioffe and C. Szegedy, 'Batch normalization: Accelerating deep network training by reducing internal covariate shift,' in International Conference on Machine Learning , 2015, pp. 448-456. -
  6. [7][7] H. Zhang, M. Ciss´ e, Y. N. Dauphin, and D. Lopez-Paz, 'mixup: Beyond empirical risk minimization,' in International Conference on Learning Representations , 2018. -
  7. [8][8] M. Denil, B. Shakibi, L. Dinh, N. De Freitas et al. , 'Predicting parameters in deep learning,' in Advances in Neural Information Processing Systems , 2013, pp. 2148-2156. -
  8. [10][10] S. Han, J. Pool, J. Tran, and W. J. Dally, 'Learning both weights and connections for efficient neural network,' in Advances in Neural Information Processing Systems , 2015, pp. 1135-1143. -
  9. [11][11] A. Polino, R. Pascanu, and D. Alistarh, 'Model compression via distillation and quantization,' in International Conference on Learning Representations , 2018. -
  10. [21][21] K. Yue, J. Deng, and F. Zhou, 'Matching guided distillation,' in European Conference on Computer Vision , 2020, pp. 312-328. -
  11. [22][22] C. Farabet, C. Couprie, L. Najman, and Y. LeCun, 'Learning hierarchical features for scene labeling,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 35, no. 8, pp. 1915-1929, 2013. -
  12. [23][23] S. Kornblith, M. Norouzi, H. Lee, and G. E. Hinton, 'Similarity of neural network representations revisited,' in International Conference on Machine Learning , vol. 97, 2019, pp. 3519-3529. -
  13. [24][24] D. Chen, J.-P. Mei, Y. Zhang, C. Wang, Z. Wang, Y. Feng, and C. Chen, 'Cross-layer distillation with semantic calibration,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2021, pp. 7028-7036. -
  14. [25][25] J. R. Hurley and R. B. Cattell, 'The procrustes program: Producing direct rotation to test a hypothesized factor structure,' Behavioral science , vol. 7, no. 2, p. 258, 1962. -
  15. [26][26] P. H. Sch¨ onemann, 'A generalized solution of the orthogonal procrustes problem,' Psychometrika , vol. 31, pp. 1-10, 1966. -
  16. [28][28] K. He, G. Gkioxari, P. Doll´ ar, and R. B. Girshick, 'Mask R-CNN,' in International Conference on Computer Vision , 2017, pp. 2980-2988. -
  17. [31][31] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, 'Xlnet: Generalized autoregressive pretraining for language understanding,' in Advances in Neural Information Processing Systems , 2019, pp. 5754-5764. -
  18. [32][32] A. Novikov, D. Podoprikhin, A. Osokin, and D. P. Vetrov, 'Tensorizing neural networks,' in Advances in Neural Information Processing Systems , 2015, pp. 442-450. -
  19. [33][33] Y. Gong, L. Liu, M. Yang, and L. Bourdev, 'Compressing deep convolutional networks using vector quantization,' in International Conference on Learning Representations , 2015. -
  20. [34][34] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, 'Pruning filters for efficient convnets,' in International Conference on Learning Representations , 2017. -
  21. [36][36] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu, 'Deep mutual learning,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 4320-4328. -
  22. [38][38] Y. Guan, P. Zhao, B. Wang, Y. Zhang, C. Yao, K. Bian, and J. Tang, 'Differentiable feature aggregation search for knowledge distillation,' in European Conference on Computer Vision , 2020, pp. 469-484. -
  23. [45][45] G. Xu, Z. Liu, X. Li, and C. C. Loy, 'Knowledge distillation meets self-supervision,' in European Conference on Computer Vision , 2020, pp. 588-604. -
  24. [46][46] S. Kullback and R. A. Leibler, 'On information and sufficiency,' The Annals of Mathematical Statistics , vol. 22, no. 1, pp. 79-86, 1951. -
  25. [47][47] M. D. Zeiler and R. Fergus, 'Visualizing and understanding convolutional networks,' in European Conference on Computer Vision , 2014, pp. 818-833. -
  26. [48][48] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning . MIT press Cambridge, 2016, vol. 1, no. 2. -
  27. [50][50] X. Wang, R. B. Girshick, A. Gupta, and K. He, 'Non-local neural networks,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 7794-7803. -
  28. [51][51] G. H. Golub and C. F. Van Loan, Matrix computations . JHU press, 2013, vol. 3. -
  29. [52][52] D. P. Bertsekas, Constrained optimization and Lagrange multiplier methods . Academic press, 2014. -
  30. [54][54] A. Coates, A. Y. Ng, and H. Lee, 'An analysis of single-layer networks in unsupervised feature learning,' in International Conference on Artificial Intelligence and Statistics , 2011, pp. 215-223. -
  31. [55][55] A. Torralba, R. Fergus, and W. T. Freeman, '80 million tiny images: A large data set for nonparametric object and scene recognition,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 30, no. 11, pp. 1958-1970, 2008. -
  32. [60][60] X. Zhang, X. Zhou, M. Lin, and J. Sun, 'Shufflenet: An extremely efficient convolutional neural network for mobile devices,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 6848-6856. -
  33. [62][62] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K¨ opf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, 'Pytorch: An imperative style, high-performance deep learning library,' in Advances in Neural Information Processing Systems , 2019, pp. 8024-8035. -
  34. [64][64] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi, 'A comprehensive overhaul of feature distillation,' in International Conference on Computer Vision , 2019, pp. 1921-1930.

Cited by

Sorted by citations

See all 335

Related

Metadata

Authors
Chen, D., Mei, J.-P., Zhang, Y., Wang, C., Wang, Z., Feng, Y., & Chen, C.
Affiliations
  • Zhejiang University (CN)
  • Zhejiang University of Technology (CN)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2021-05-18
Volume
35
Issue
8
Pages
7028-7036
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v35i8.16865
OpenAlex
W3174102142
Cited by
335
References
29
Citations by year
2026: 25 · 2025: 88 · 2024: 92 · 2023: 79 · 2022: 38 · 2021: 13
FWCI
18.15
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Funders
State Grid Corporation of China
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
semantic calibration, attention mechanism, teacher-student models, semantic mismatch, neural network compression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review