Cross-Layer Distillation with Semantic Calibration
Open accessAbstract
Recently proposed knowledge distillation approaches based on feature-map transfer validate that intermediate layers of a teacher model can serve as effective targets for training a student model to obtain better generalization ability. Existing studies mainly focus on particular representation forms for knowledge transfer between manually specified pairs of teacher-student intermediate layers. However, semantics of intermediate layers may vary in different networks and manual association of layers might lead to negative regularization caused by semantic mismatch between certain teacher-student layer pairs. To address this problem, we propose Semantic Calibration for Cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple layers rather than a single fixed intermediate layer from the teacher model for appropriate cross-layer supervision in training. Consistent improvements over state-of-the-art approaches are observed in extensive experiments with various network architectures for teacher and student models, demonstrating the effectiveness and flexibility of the proposed attention based soft layer association mechanism for cross-layer distillation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The authors' approach enables greater compression of the teacher model compared to previous methods that relied on pruning, knowledge distillation, or both.
“Our approach provides a clear advantage by enabling a greater compression of the teacher model compared to the previous methods that relied on pruning [10-15], knowledge distillation (KD) [1,3,5,7,8], and the combination of both [16-20].”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · see citing works
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 4× · see citing works
“This problem becomes especially severe for random selection (FitNet method fails in 4/7 cases) and the situation where the number of candidate layers sL is larger than tL (4/5 of methods fail in the “ShuffleNetV2 & VGG-13” setting).”
MethodInfluentialZagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244 · see citing works
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859 · see citing works
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · see citing works
Zagoruyko, S., & Komodakis, N. (2016) · arXiv · doi:10.48550/arxiv.1612.03928 · Cited by 2,027 · see citing works
Relational Knowledge Distillation
Park, W., Kim, D., Lu, Y., & Cho, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00409 · Cited by 1,783 · see citing works
Do Deep Nets Really Need to be Deep?Preprint
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,592 · see citing works
Similarity-Preserving Knowledge Distillation
Tung, F., & Mori, G. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00145 · Cited by 1,179 · see citing works
Contrastive Representation DistillationPreprint
Tian, Y., Krishnan, D., & Isola, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10699 · Cited by 888 · see citing works
When does label smoothing help?
Müller, R., Kornblith, S., & Hinton, G. (2019) · arXiv · doi:10.48550/arxiv.1906.02629 · Cited by 859 · see citing works
Variational Information Distillation for Knowledge Transfer
Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., & Dai, Z. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00938 · Cited by 710 · see citing works
Regularizing Neural Networks by Penalizing Confident Output Distributions
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1701.06548 · Cited by 655 · see citing works
Correlation Congruence for Knowledge Distillation
Peng, B., Xiao, J. J., Li, D., Zhou, S., Wu, Y., Liu, J., Zhang, Z., & Liu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00511 · Cited by 608 · see citing works
Revisiting Knowledge Distillation via Label Smoothing Regularization
Yuan, L., Tay, F. E., Li, G., Wang, T., & Feng, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00396 · Cited by 572 · see citing works
Learning Deep Representations with Probabilistic Knowledge Transfer
Passalis, N., & Tefas, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_17 · Cited by 488 · see citing works
Knowledge Distillation via Instance Relationship Graph
Liu, Y., Cao, J., Li, B., Yuan, C., Hu, W., Li, Y., & Duan, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00726 · Cited by 325 · see citing works
Online Knowledge Distillation with Diverse Peers
Chen, D., Mei, J.-P., Wang, C., Feng, Y., & Chen, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5746 · Cited by 324 · see citing works
Self-citationLarge scale distributed neural network training through online distillationPreprint
Anil, R., Pereyra, G., Passos, A., Ormándi, R., Dahl, G. E., & Hinton, G. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.03235 · Cited by 315 · see citing works
Heterogeneous Knowledge Distillation Using Information Flow Modeling
Passalis, N., Tzelepi, M., & Tefas, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00241 · Cited by 156 · see citing works
Learning What and Where to TransferPreprint
Jang, Y., Lee, H., Hwang, S. J., & Shin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.05901 · Cited by 113 · see citing works
Flagged first, then by citations
Research works printed but not linked: 35
- [2][2] S. Han, H. Mao, and W. J. Dally, 'Deep compression: Compressing deep neural network with pruning, trained quantization and huffmancoding,' in International Conference on Learning Representations , 2016. -
- [3][3] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, 'Understanding deep learning requires rethinking generalization,' in International Conference on Learning Representations , 2017. -
- [4][4] K. He, X. Zhang, S. Ren, and J. Sun, 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,' in International Conference on Computer Vision , 2015, pp. 1026-1034. -
- [5][5] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, 'Dropout: a simple way to prevent neural networks from overfitting,' Journal of Machine Learning Researh , vol. 15, no. 1, pp. 1929-1958, 2014. -
- [6][6] S. Ioffe and C. Szegedy, 'Batch normalization: Accelerating deep network training by reducing internal covariate shift,' in International Conference on Machine Learning , 2015, pp. 448-456. -
- [7][7] H. Zhang, M. Ciss´ e, Y. N. Dauphin, and D. Lopez-Paz, 'mixup: Beyond empirical risk minimization,' in International Conference on Learning Representations , 2018. -
- [8][8] M. Denil, B. Shakibi, L. Dinh, N. De Freitas et al. , 'Predicting parameters in deep learning,' in Advances in Neural Information Processing Systems , 2013, pp. 2148-2156. -
- [10][10] S. Han, J. Pool, J. Tran, and W. J. Dally, 'Learning both weights and connections for efficient neural network,' in Advances in Neural Information Processing Systems , 2015, pp. 1135-1143. -
- [11][11] A. Polino, R. Pascanu, and D. Alistarh, 'Model compression via distillation and quantization,' in International Conference on Learning Representations , 2018. -
- [21][21] K. Yue, J. Deng, and F. Zhou, 'Matching guided distillation,' in European Conference on Computer Vision , 2020, pp. 312-328. -
- [22][22] C. Farabet, C. Couprie, L. Najman, and Y. LeCun, 'Learning hierarchical features for scene labeling,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 35, no. 8, pp. 1915-1929, 2013. -
- [23][23] S. Kornblith, M. Norouzi, H. Lee, and G. E. Hinton, 'Similarity of neural network representations revisited,' in International Conference on Machine Learning , vol. 97, 2019, pp. 3519-3529. -
- [24][24] D. Chen, J.-P. Mei, Y. Zhang, C. Wang, Z. Wang, Y. Feng, and C. Chen, 'Cross-layer distillation with semantic calibration,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2021, pp. 7028-7036. -
- [25][25] J. R. Hurley and R. B. Cattell, 'The procrustes program: Producing direct rotation to test a hypothesized factor structure,' Behavioral science , vol. 7, no. 2, p. 258, 1962. -
- [26][26] P. H. Sch¨ onemann, 'A generalized solution of the orthogonal procrustes problem,' Psychometrika , vol. 31, pp. 1-10, 1966. -
- [28][28] K. He, G. Gkioxari, P. Doll´ ar, and R. B. Girshick, 'Mask R-CNN,' in International Conference on Computer Vision , 2017, pp. 2980-2988. -
- [29][29] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton et al. , 'Mastering the game of go without human knowledge,' Nature , vol. 550, no. 7676, pp. 354-359, 2017. -
- [31][31] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, 'Xlnet: Generalized autoregressive pretraining for language understanding,' in Advances in Neural Information Processing Systems , 2019, pp. 5754-5764. -
- [32][32] A. Novikov, D. Podoprikhin, A. Osokin, and D. P. Vetrov, 'Tensorizing neural networks,' in Advances in Neural Information Processing Systems , 2015, pp. 442-450. -
- [33][33] Y. Gong, L. Liu, M. Yang, and L. Bourdev, 'Compressing deep convolutional networks using vector quantization,' in International Conference on Learning Representations , 2015. -
- [34][34] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, 'Pruning filters for efficient convnets,' in International Conference on Learning Representations , 2017. -
- [36][36] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu, 'Deep mutual learning,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 4320-4328. -
- [38][38] Y. Guan, P. Zhao, B. Wang, Y. Zhang, C. Yao, K. Bian, and J. Tang, 'Differentiable feature aggregation search for knowledge distillation,' in European Conference on Computer Vision , 2020, pp. 469-484. -
- [45][45] G. Xu, Z. Liu, X. Li, and C. C. Loy, 'Knowledge distillation meets self-supervision,' in European Conference on Computer Vision , 2020, pp. 588-604. -
- [46][46] S. Kullback and R. A. Leibler, 'On information and sufficiency,' The Annals of Mathematical Statistics , vol. 22, no. 1, pp. 79-86, 1951. -
- [47][47] M. D. Zeiler and R. Fergus, 'Visualizing and understanding convolutional networks,' in European Conference on Computer Vision , 2014, pp. 818-833. -
- [48][48] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning . MIT press Cambridge, 2016, vol. 1, no. 2. -
- [50][50] X. Wang, R. B. Girshick, A. Gupta, and K. He, 'Non-local neural networks,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 7794-7803. -
- [51][51] G. H. Golub and C. F. Van Loan, Matrix computations . JHU press, 2013, vol. 3. -
- [52][52] D. P. Bertsekas, Constrained optimization and Lagrange multiplier methods . Academic press, 2014. -
- [54][54] A. Coates, A. Y. Ng, and H. Lee, 'An analysis of single-layer networks in unsupervised feature learning,' in International Conference on Artificial Intelligence and Statistics , 2011, pp. 215-223. -
- [55][55] A. Torralba, R. Fergus, and W. T. Freeman, '80 million tiny images: A large data set for nonparametric object and scene recognition,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 30, no. 11, pp. 1958-1970, 2008. -
- [60][60] X. Zhang, X. Zhou, M. Lin, and J. Sun, 'Shufflenet: An extremely efficient convolutional neural network for mobile devices,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 6848-6856. -
- [62][62] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K¨ opf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, 'Pytorch: An imperative style, high-performance deep learning library,' in Advances in Neural Information Processing Systems , 2019, pp. 8024-8035. -
- [64][64] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi, 'A comprehensive overhaul of feature distillation,' in International Conference on Computer Vision , 2019, pp. 1921-1930.
Cited by
Sorted by citations
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931 · see citing works
Lightweight Deep Learning for Resource-Constrained Environments: A Survey
Liu, H.-I., Galindo, M., Xie, H., Wong, L.-K., Shuai, H.-H., Li, Y., & Cheng, W.-H. (2024) · ACM Computing Surveys · doi:10.1145/3657282 · Cited by 296 · see citing works
Knowledge Distillation with the Reused Teacher Classifier
Chen, D., Mei, J.-P., Zhang, H., Wang, C., Feng, Y., & Chen, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01163 · Cited by 254 · see citing works
Self-citationEdge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI
Yao, J., Zhang, S., Yao, Y., Wang, F., Ma, J., Zhang, J., Chu, Y., Ji, L., Jia, K., Shen, T., et al. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178211 · Cited by 156 · see citing works
Knowledge Distillation via the Target-aware Transformer
Lin, S., Xie, H., Wang, B., Yu, K., Chang, X., Liang, X., & Wang, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01064 · Cited by 148 · see citing works
Confidence-Aware Multi-Teacher Knowledge Distillation
Zhang, H., Chen, D., & Wang, C. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747534 · Cited by 142 · see citing works
Self-citationExploring Inter-Channel Correlation for Diversity-preserved Knowledge Distillation
Liu, L., Huang, Q., Lin, S., Xie, H., Wang, B., Chang, X., & Liang, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00816 · Cited by 134 · see citing works
Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition
Yang, C., An, Z., Zhou, H., Zhuang, F., Xu, Y., & Zhang, Q. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3257878 · Cited by 114 · see citing works
Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learning
Huang, W., Ye, M., Shi, Z., & Du, B. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327373 · Cited by 83 · see citing works
Distilling a Powerful Student Model via Online Knowledge Distillation
Li, S., Lin, M., Wang, Y., Wu, Y., Tian, Y., Shao, L., & Ji, R. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3152732 · Cited by 75 · see citing works
Distilling Holistic Knowledge with Graph Neural Networks
Zhou, S., Wang, Y., Chen, D., Chen, J., Wang, X., Wang, C., & Bu, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01022 · Cited by 72 · see citing works
Self-citationEnabling All In-Edge Deep Learning: A Literature Review
Joshi, P., Hasanuzzaman, M., Thapa, C., Afli, H., & Scully, T. (2023) · IEEE Access · doi:10.1109/access.2023.3234761 · Cited by 60 · see citing works
Wei, S., Luo, C., & Luo, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01512 · Cited by 54 · see citing works
SelfReg-UNet: Self-Regularized UNet for Medical Image Segmentation
Zhu, W., Chen, X., Qiu, P., Farazi, M., Sotiras, A., Razi, A., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72111-3_56 · Cited by 53 · see citing works
A Survey on Symbolic Knowledge Distillation of Large Language Models
Acharya, K., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3428519 · Cited by 50 · see citing works
CDFKD-MFS: Collaborative Data-Free Knowledge Distillation via Multi-Level Feature Sharing
Hao, Z., Luo, Y., Wang, Z., Hu, H., & An, J. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3192663 · Cited by 45 · see citing works
Self-citationZhao, Q., Zhong, L., Xiao, J., Zhang, J., Chen, Y., Liao, W., Zhang, S., & Wang, G. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3262680 · Cited by 45 · see citing works
Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation
Yang, C., Yu, X., An, Z., & Xu, Y. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-32095-8_1 · Cited by 43 · see citing works
Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning
Song, K., Xie, J., Zhang, S., & Luo, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01140 · Cited by 43 · see citing works
He, R., Sun, S., Yang, J., Bai, S., & Qi, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00895 · Cited by 40 · see citing works
Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning
Zhang, H., Chen, D., & Wang, C. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00333 · Cited by 40 · see citing works
Self-citationCollaborative Knowledge Distillation for Heterogeneous Information Network Embedding
Wang, C., Zhou, S., Yu, K., Chen, D., Li, B., Feng, Y., & Chen, C. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512209 · Cited by 39 · see citing works
Self-citationTAKD: Target-Aware Knowledge Distillation for Remote Sensing Scene Classification
Wu, J., Fang, L., & Yue, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3391018 · Cited by 37 · see citing works
Xie, L., Cen, X., Lu, H., Yin, G., & Yin, M. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102526 · Cited by 35 · see citing works
Applications of knowledge distillation in remote sensing: A survey
Himeur, Y., Aburaed, N., Elharrouss, O., Varlamis, I., Atalla, S., Mansoor, W., & Al-Ahmad, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102742 · Cited by 35 · see citing works
FedHe: Heterogeneous Models and Communication-Efficient Federated Learning
Chan, Y.-H., & Ngai, E. C. ‐. (2021) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn53354.2021.00043 · Cited by 34 · see citing works
SemCKD: Semantic Calibration for Cross-Layer Knowledge Distillation
Wang, C., Chen, D., Mei, J.-P., Zhang, Y., Feng, Y., & Chen, C. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3171571 · Cited by 36 · see citing works
Self-citationDiversified branch fusion for self-knowledge distillation
Long, Z., Ma, F., Sun, B., Tan, M., & Shutao Li, A. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.09.007 · Cited by 34 · see citing works
Zhang, X., Zhu, J., Wang, D., Wang, Y.-Y., Liang, T., Wang, H., & Yin, Y. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111762 · Cited by 33 · see citing works
Contrastive Correlation Preserving Replay for Online Continual Learning
Yu, D.-P., Zhang, M., Li, M., Zha, F., Zhang, J., Sun, L., & Huang, K. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3285221 · Cited by 33 · see citing works
Robust Saliency-Aware Distillation for Few-Shot Fine-Grained Visual Recognition
Liu, H., Chen, C. L. P., Gong, X., & Zhang, T. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3369870 · Cited by 31 · see citing works
Self-citationDeep Cross-Layer Collaborative Learning Network for Online Knowledge Distillation
Su, T., Liang, Q., Zhang, J., Yu, Z., Xu, Z., Wang, G., & Liu, X. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3222013 · Cited by 31 · see citing works
Ding, X., Wang, Y., Xu, Z., Wang, Z. J., & Welch, W. J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119060 · Cited by 29 · see citing works
Self-citationZhou, W., Jian, B., Fang, M., Dong, X., Liu, Y., & Jiang, Q. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3382354 · Cited by 29 · see citing works
Divide and Distill: New Outlooks on Knowledge Distillation for Environmental Sound Classification
Tripathi, A. M., & Pandey, O. J. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3244507 · Cited by 30 · see citing works
Low-Resolution Object Recognition With Cross-Resolution Relational Contrastive Distillation
Zhang, K., Ge, S., Shi, R., & Zeng, D. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3310042 · Cited by 28 · see citing works
Zhang, J., Huang, Z., & Ohn-Bar, E. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00754 · Cited by 27 · see citing works
Data Distribution Distilled Generative Model for Generalized Zero-Shot Recognition
Wang, Y., Hong, M., Huangfu, L., & Huang, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28381 · Cited by 26 · see citing works
Semantic-aware knowledge distillation with parameter-free feature uniformization
Guo, G., Han, L., Wang, L., Zhang, D., & Han, J. (2023) · Visual Intelligence · doi:10.1007/s44267-023-00003-0 · Cited by 26 · see citing works
Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution
Yang, C., An, Z., Cai, L., & Xu, Y. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3186807 · Cited by 24 · see citing works
Cross-Layer Contrastive Learning of Latent Semantics for Facial Expression Recognition
Xie, W., Peng, Z., Shen, L., Lu, W., Zhang, Y., & Song, S. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3378459 · Cited by 22 · see citing works
Self-citationKnowledge Distillation in Histology Landscape by Multi-Layer Features Supervision
Javed, S., Mahmood, A., Qaiser, T., & Werghi, N. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3237749 · Cited by 24 · see citing works
Attention-Based Knowledge Distillation in Scene Recognition: The Impact of a DCT-Driven Loss
López-Cifuentes, A., Escudero-Viñolo, M., Bescós, J., & SanMiguel, J. C. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3250031 · Cited by 23 · see citing works
A Decade of Video Analytics at Edge: Training, Deployment, Orchestration, and Platforms
Bai, T., Zhao, H., Huang, L., Wang, Z., Kim, D. I., & Nallanathan, A. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3563377 · Cited by 22 · see citing works
Self-citationImproving SAR Automatic Target Recognition via Trusted Knowledge Distillation From Simulated Data
Han, F., Dong, H., Si, L., & Zhang, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3360470 · Cited by 21 · see citing works
Light-M: An efficient lightweight medical image segmentation framework for resource-constrained IoMT
Zhang, Y., Chen, Z., & Yang, X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108088 · Cited by 21 · see citing works
Self-citationKnowledge distillation via adaptive meta-learning for graph neural network
Shen, T., Wang, J., & Zhang, X. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121505 · Cited by 21 · see citing works
Rethinking Knowledge Distillation with Raw Features for Semantic Segmentation
Liu, T., Chen, C., Yang, X., & Tan, W. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00119 · Cited by 20 · see citing works
Self-citationFine-Grained Learning Behavior-Oriented Knowledge Distillation for Graph Neural Networks
Liu, K., Huang, Z., Wang, C., Gao, B., & Chen, Y. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3420895 · Cited by 20 · see citing works
Self-citationMulti-View mutual learning network for multimodal fake news detection
Cui, W., Zhang, X., & Shang, M. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127407 · Cited by 21 · see citing works
Online adversarial knowledge distillation for graph neural networks
Wang, C., Wang, Z., Chen, D., Zhou, S., Feng, Y., & Chen, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121671 · Cited by 20 · see citing works
Self-citationEfficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning
Malihi, L., & Heidemann, G. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7030154 · Cited by 20 · see citing works
Teacher-free Distillation via Regularizing Intermediate Representation
Li, L., Liang, S.-N., Yang, Y., & Jin, Z. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892575 · Cited by 19 · see citing works
Umirzakova, S., Abdullaev, M., Mardieva, S., Мухтаржановна, Л. Н., & Muksimova, S. (2024) · Electronics · doi:10.3390/electronics13224530 · Cited by 19 · see citing works
Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation
Lu, A., Zhao, J., Li, C., Xiao, Y., & Luo, B. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680878 · Cited by 18 · see citing works
Attention-based cross-layer domain alignment for unsupervised domain adaptation
Ma, X., Yuan, J., Chen, Y., Tong, R., & Lin, L. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.086 · Cited by 18 · see citing works
Multi-Scale Distillation from Multiple Graph Neural Networks
Zhang, C., Liu, J., Dang, K., & Zhang, W. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20354 · Cited by 18 · see citing works
Improving Knowledge Distillation With a Customized Teacher
Tan, C., & Liu, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3189680 · Cited by 18 · see citing works
AMD: Automatic Multi-step Distillation of Large-Scale Vision Models
Han, C., Wang, Q., Dianat, S. A., Rabbani, M., Rao, R., Fang, Y. X., Guan, Q., Huang, L., & Liu, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73650-6_25 · Cited by 18 · see citing works
Sequence-aware Knowledge Distillation for a Lightweight Event Representation
Zheng, J., Cai, F., Ling, Y., & Chen, H. (2022) · ACM Transactions on Information Systems · doi:10.1145/3545798 · Cited by 17 · see citing works
MHKD-MVQA: Multimodal Hierarchical Knowledge Distillation for Medical Visual Question Answering
Wang, J., Huang, S., Du, H., Qin, Y. X., Wang, H., & Zhang, W. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995473 · Cited by 17 · see citing works
DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving
He, J.-Y., Cheng, Z.-Q., Li, C., Xiang, W., Chen, B., Luo, B., Geng, Y., & Xie, X. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/90 · Cited by 17 · see citing works
Collaborative knowledge distillation via filter knowledge transfer
Gou, J., Hu, Y., Sun, L., Wang, Z., & Ma, H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121884 · Cited by 17 · see citing works
Self-citationExploring the Knowledge Transferred by Response-Based Teacher-Student Distillation
Song, L., Gong, X., Zhou, H., Chen, J., Zhang, Q., Doermann, D., & Yuan, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612162 · Cited by 17 · see citing works
Learning structure perception MLPs on graphs: a layer-wise graph knowledge distillation framework
Du, H., Yu, R., Bai, L., Bai, L., & Wang, W. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02150-2 · Cited by 16 · see citing works
Zhang, H., Liu, L., Huang, Y., Yang, Z., Lei, X., & Wen, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01452 · Cited by 16 · see citing works
Channel-Correlation-Based Selective Knowledge Distillation
Gou, J., Xiong, X., Yu, B., Zhan, Y., & Zhang, Y. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3232569 · Cited by 15 · see citing works
Self-citationA Lightweight Pig Face Recognition Method Based on Automatic Detection and Knowledge Distillation
Ma, R., Ali, H. O., Chung, S., Kim, S. C., & Kim, H. (2023) · Applied Sciences · doi:10.3390/app14010259 · Cited by 15 · see citing works
Zhang, J., Zhang, Y., Shao, F., Ma, X., Feng, S., Wu, Y., & Zhou, D. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112237 · Cited by 15 · see citing works
Self-citationDominguez-Morales, J. P., Hernández‐Rodríguez, J., Durán-López, L., Conejo‐Mir, J., & Rodríguez, J. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3465929 · Cited by 15 · see citing works
Zhou, W., Jian, B., Liu, Y., & Jiang, Q. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3575088 · Cited by 14 · see citing works
Zhou, W., Xie, J., & Xu, C. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3675381 · Cited by 14 · see citing works
Extreme Low-Resolution Action Recognition with Confident Spatial-Temporal Attention Transfer
Bai, Y., Zou, Q., Chen, X., Li, L., Ding, Z., & Chen, L. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01771-4 · Cited by 14 · see citing works
Yang, M., Chengke, W., Guo, Y., He, Y. Z., Jiang, R., Jiang, J., & Yang, Z. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102294 · Cited by 14 · see citing works
ATMKD: adaptive temperature guided multi-teacher knowledge distillation
Lin, Y., Yin, S., Ding, Y., & Liang, X. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01483-w · Cited by 14 · see citing works
Direct Distillation Between Different Domains
Tang, J., Chen, S., Niu, G., Zhu, H., Zhou, J. T., Gong, C., & Sugiyama, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_9 · Cited by 14 · see citing works
I-YOLOv11n: A Lightweight and Efficient Small Target Detection Framework for UAV Aerial Images
Ma, Y., Xi, C., Ma, T., Sun, H. X., Lu, H., Xu, X., & Chen, X. (2025) · Sensors · doi:10.3390/s25154857 · Cited by 13 · see citing works
SCL-IKD: intermediate knowledge distillation via supervised contrastive representation learning
Sharma, S., Lodhi, S. S., & Chandra, J. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05036-y · Cited by 13 · see citing works
Huang, X.-Y., Chen, S.-Y., & Wei, C.-S. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3335943 · Cited by 13 · see citing works
Cross-View Consistency Regularisation for Knowledge Distillation
Zhang, W., Liu, D., Cai, W., & Ma, C. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681206 · Cited by 13 · see citing works
Ability-aware knowledge distillation for resource-constrained embedded devices
Xiong, Y., Zhai, W., Xu, X., Wang, J., Zhu, Z., Ji, C., & Cao, J. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102912 · Cited by 12 · see citing works
Cross-Architecture Distillation for Face Recognition
Zhao, W., Zhu, X., He, Z., Zhang, X., & Lei, Z. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611711 · Cited by 12 · see citing works
Balanced self-distillation for long-tailed recognition
Ren, N., Li, X., Wu, Y., & Fu, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111504 · Cited by 12 · see citing works
Yang, G., Tang, Y., Wu, Z., Li, J., Xu, J., & Wan, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446978 · Cited by 12 · see citing works
Decoupled Knowledge Distillation via Spatial Feature Blurring for Hyperspectral Image Classification
Xie, W., Zhang, Z., Jiao, L., & Wang, J. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3383854 · Cited by 12 · see citing works
Matching the Ideal Pruning Method with Knowledge Distillation for Optimal Compression
Malihi, L., & Heidemann, G. (2024) · Applied System Innovation · doi:10.3390/asi7040056 · Cited by 12 · see citing works
Wang, K., Zheng, F., Guan, D., Liu, J., & Qin, J. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111173 · Cited by 12 · see citing works
Improve Knowledge Distillation via Label Revision and Data Selection
Lan, W., Cheung, Y., Xu, Q., Liu, B., Hu, Z., Li, M., & Chen, Z. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2025.3559881 · Cited by 12 · see citing works
Co-Completion for Occluded Facial Expression Recognition
Xing, Z., Tan, W., He, R., Lin, Y., & Yan, B. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548183 · Cited by 12 · see citing works
Zhang, Y., Yan, Z., Sun, X., Lü, X., Li, J., Mao, Y., & Wang, L. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3241342 · Cited by 12 · see citing works
Self-citationAudio Representation Learning by Distilling Video as Privileged Information
Hajavi, A., & Etemad, A. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3243596 · Cited by 11 · see citing works
TIE-KD: Teacher-independent and explainable knowledge distillation for monocular depth estimation
Choi, S.-W., Choi, D., & Kim, D. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105110 · Cited by 11 · see citing works
Learning Student Network Under Universal Label Noise
Tang, J., Jiang, N., Zhu, H., Zhou, J. T., & Gong, C. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3430539 · Cited by 11 · see citing works
Difficulty level-based knowledge distillation
Ham, G., Cho, Y., Lee, J.-H., Kang, M., Choi, G., & Kim, D. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128375 · Cited by 11 · see citing works
MLink: Linking Black-Box Models for Collaborative Multi-Model Inference
Yuan, M., Zhang, L., & Li, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i9.21180 · Cited by 11 · see citing works
Yu, C.-C., Chen, T.-Y., Hsu, C.-W., & Cheng, H.-Y. (2024) · Electronics · doi:10.3390/electronics13030583 · Cited by 10 · see citing works
Multi-Time Knowledge Distillation
Chen, G., Fan, Z., Zhu, Y., & Zhang, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129377 · Cited by 10 · see citing works
DenseKD: Dense Knowledge Distillation by Exploiting Region and Sample Importance
Zhang, H., Liu, L., Zhang, Y., Lei, X., Hui, F., & Wen, B. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3525737 · Cited by 10 · see citing works
Self-citationUnion-Domain Knowledge Distillation for Underwater Acoustic Target Recognition
Chu, X., Duan, H., Wen, Z., Xu, L., Hu, R., & Xiang, W. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3539476 · Cited by 10 · see citing works
Zhang, Y., Zhang, W., Li, J., Qi, X., Lü, X., Wang, L., & Hou, Y. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.05.023 · Cited by 10 · see citing works
Self-citation
- Authors
- Chen, D., Mei, J.-P., Zhang, Y., Wang, C., Wang, Z., Feng, Y., & Chen, C.
- Affiliations
- Zhejiang University (CN)
- Zhejiang University of Technology (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2021-05-18
- Volume
- 35
- Issue
- 8
- Pages
- 7028-7036
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W3174102142
- Cited by
- 335
- References
- 29
- Citations by year
- 2026: 25 · 2025: 88 · 2024: 92 · 2023: 79 · 2022: 38 · 2021: 13
- FWCI
- 18.15
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- State Grid Corporation of China
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- semantic calibration, attention mechanism, teacher-student models, semantic mismatch, neural network compression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
