The DOI matches its Crossref record; 152 of the 152 references it cites were found in a registry or an index.
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
Open accessAbstract
Deep neural models, in recent years, have been successful in almost every field, even solving the most complex problem statements. However, these models are huge in size with millions (and even billions) of parameters, demanding heavy computation power and failing to be deployed on edge devices. Besides, the performance boost is highly dependent on redundant labeled data. To achieve faster speeds and to handle the problems caused by the lack of labeled data, knowledge distillation (KD) has been proposed to transfer information learned from one model to another. KD is often characterized by the so-called 'Student-Teacher' (S-T) learning framework and has been broadly applied in model compression and knowledge transfer. This paper is about KD and S-T learning, which are being actively studied in recent years. First, we aim to provide explanations of what KD is and how/why it works. Then, we provide a comprehensive survey on the recent progress of KD methods together with S-T frameworks typically used for vision tasks. In general, we investigate some fundamental questions that have been driving this research area and thoroughly generalize the research progress and technical details. Additionally, we systematically analyze the research status of KD in vision applications. Finally, we discuss the potentials and open challenges of existing methods and prospect the future directions of KD and S-T learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Inductive Representation Learning on Large Graphs
Hamilton, W. L., Ying, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1706.02216 · Cited by 10,639
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Graph neural networks: A review of methods and applications
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., & Sun, M. (2020) · AI Open · doi:10.1016/j.aiopen.2021.01.001 · Cited by 6,880
Self-citationBengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“[131] generalize the selection of distillation positions by employing variational information maximization [18], curriculum learning [20], adversarial learning [65], similarity-presentation in representation learning [91], muti-task learning [43], reinforcement learning [163].”
MethodBootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 3×
“examples, BYOL [178] directly bootstraps the representations with two neural networks, referred to as online and target networks, that interact and learn from each other.”
BackgroundInfluentialFitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A. A., & Darrell, T. (2017) · arXiv · doi:10.48550/arxiv.1711.03213 · Cited by 2,656 · cited 2×
“Some methods such as [278, 280] distill the knowledge from the source domain to the target domain based on adversarial learning [78] and image translation [123, 124, 200].”
MethodMeasurement Reliability and Validity
(2000) · Research Methods in Applied Settings · doi:10.4324/9781410605337-29 · Cited by 2,541
Self-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications
Cai, H., Zheng, V. W., & Chang, K. C.-C. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2807452 · Cited by 2,034
Attention gated networks: learning to leverage salient regions in medical images
Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.012 · Cited by 2,014
Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00454 · Cited by 1,752
Relational Knowledge Distillation
Park, W., Kim, D., Lu, Y., & Cho, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00409 · Cited by 1,681
A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning
Yim, J., Joo, D., Bae, J., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.754 · Cited by 1,570
Learning a Unified Classifier Incrementally via Rebalancing
Hou, S., Pan, X., Loy, C. C., Wang, Z., & Lin, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00092 · Cited by 1,195
End-to-End Incremental Learning
Castro, F. M., Marín-Jiménez, M. J., Guil, N., Schmid, C., & Alahari, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01258-8_15 · Cited by 1,184
Multi-Task Deep Neural Networks for Natural Language Understanding
Liu, X., He, P., Chen, W., & Gao, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1441 · Cited by 1,154
Similarity-Preserving Knowledge Distillation
Tung, F., & Mori, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00145 · Cited by 1,079
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., & Ma, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00381 · Cited by 954
To prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805
Arandjelovic, R., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 782
A Comprehensive Overhaul of Feature Distillation
Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., & Choi, J. Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00201 · Cited by 706
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703
Structured Knowledge Distillation for Semantic Segmentation
Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., & Wang, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00271 · Cited by 681
Variational Information Distillation for Knowledge Transfer
Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., & Dai, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00938 · Cited by 671
Through-Wall Human Pose Estimation Using Radio Signals
Zhao, M., Li, T., Alsheikh, M. A., Tian, Y., Zhao, H., Torralba, A., & Katabi, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00768 · Cited by 658
On the Efficacy of Knowledge Distillation
Cho, J. H., & Hariharan, B. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00489 · Cited by 637
Patient Knowledge Distillation for BERT Model Compression
Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1441 · Cited by 631
Correlation Congruence for Knowledge Distillation
Peng, B., Jin, X., Li, D., Zhou, S., Wu, Y., Liu, J., Zhang, Z., & Liu, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00511 · Cited by 570
Learning from Noisy Labels with Distillation
Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., & Li, L.-J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.211 · Cited by 541
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion
Yin, H., Molchanov, P., Alvarez, J. M., Li, Z., Mallya, A., Hoiem, D., Jha, N. K., & Kautz, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00874 · Cited by 540
Cross Modal Distillation for Supervision Transfer
Gupta, S., Hoffman, J., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.309 · Cited by 530
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
Heo, B., Lee, M., Yun, S., & Choi, J. Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013779 · Cited by 522
Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
Papernot, N., Abadi, M., Erlingsson, Ú., Goodfellow, I., & Talwar, K. (2016) · arXiv · doi:10.48550/arxiv.1610.05755 · Cited by 492
Distilling Object Detectors With Fine-Grained Feature Imitation
Wang, T., Yuan, L., Zhang, X., & Feng, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00507 · Cited by 452
Data Distillation: Towards Omni-Supervised Learning
Radosavovic, I., Dollar, P., Girshick, R., Gkioxari, G., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00433 · Cited by 421
Learning from Multiple Teacher Networks
You, S., Xu, C., Xu, C., & Tao, D. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098135 · Cited by 395
Ambient Sound Provides Supervision for Visual Learning
Owens, A., Wu, J., McDermott, J. H., Freeman, W. T., & Torralba, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_48 · Cited by 373
Exploring Object Relation in Mean Teacher for Cross-Domain Detection
Cai, Q., Pan, Y., Ngo, C.-W., Tian, X., Duan, L., & Yao, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01172 · Cited by 347
Data-Free Learning of Student Networks
Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu, C., Xu, C., & Tian, Q. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00361 · Cited by 346
Knowledge Distillation by On-the-Fly Native Ensemble
Xu, L., Zhu, X., & Gong, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04606 · Cited by 339
Online Knowledge Distillation with Diverse Peers
Chen, D., Mei, J.-P., Wang, C., Feng, Y., & Chen, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5746 · Cited by 324
Knowledge Distillation via Instance Relationship Graph
Liu, Y., Cao, J., Li, B., Yuan, C., Hu, W., Li, Y., & Duan, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00726 · Cited by 303
Boosting Self-Supervised Learning via Knowledge Transfer
Noroozi, M., Vinjimoor, A., Favaro, P., & Pirsiavash, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00975 · Cited by 296
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
Albanie, S., Nagrani, A., Vedaldi, A., & Zisserman, A. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240578 · Cited by 290
Dual Student: Breaking the Limits of the Teacher in Semi-Supervised Learning
Ke, Z., Wang, D., Yan, Q., Ren, J., & Lau, R. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00683 · Cited by 271
Unifying distillation and privileged information
Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V. (2015) · arXiv · doi:10.48550/arxiv.1511.03643 · Cited by 270 · cited 1×
“[10] propose a generalized distillation DA method by applying the generalized distillation information [144] to multiple teachers to generate soft labels which are then used to supervise the student model (this framework is similar to online KD from multiple teachers as mentioned in Sec.”
MethodLearning From Multiple Experts: Self-paced Knowledge Distillation for Long-Tailed Classification
Xiang, L., Ding, G., & Han, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58558-7_15 · Cited by 268
Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning
Luo, Y., Zhu, J., Li, M., Ren, Y., & Zhang, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00927 · Cited by 264
Knowledge Distillation Meets Self-supervision
Xu, G., Liu, Z., Li, X., & Loy, C. C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58545-7_34 · Cited by 264
Self-Ensembling With GAN-Based Data Augmentation for Domain Adaptation in Semantic Segmentation
Choi, J., Kim, T., & Kim, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00693 · Cited by 248
Efficient Knowledge Distillation from an Ensemble of Teachers
Fukuda, T., Suzuki, M., Kurata, G., Thomas, S., Cui, J., & Ramabhadran, B. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-614 · Cited by 243
Training Deep Neural Networks in Generations: A More Tolerant Teacher Educates Better Students
Yang, C., Xie, L., Qiao, S., & Yuille, A. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015628 · Cited by 234
Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching
Nagrani, A., Albanie, S., & Zisserman, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00879 · Cited by 220
GAN Compression: Efficient Architectures for Interactive Conditional GANs
Li, M., Lin, J., Ding, Y., Liu, Z., Zhu, J.-Y., & Han, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00533 · Cited by 216
Unpaired Multi-Modal Segmentation via Knowledge Distillation
Dou, Q., Liu, Q., Heng, P. A., & Glocker, B. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2963882 · Cited by 212
Learning Monocular Depth by Distilling Cross-Domain Stereo Networks
Guo, X., Li, H., Yi, S., Ren, J., & Wang, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_30 · Cited by 208
BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Clark, K., Luong, M.-T., Khandelwal, U., Manning, C. D., & Le, Q. V. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1595 · Cited by 208
Snapshot Distillation: Teacher-Student Optimization in One Generation
Yang, C., Xie, L., Su, C., & Yuille, A. L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00297 · Cited by 208
Triple Generative Adversarial Nets
Li, C., Xu, K., Zhu, J., & Zhang, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.02291 · Cited by 204
Wang, L., Mostafavi, I. S. M., Ho, Y.-S., & Yoon, K.-J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01032 · Cited by 195
Self-citationDiversity With Cooperation: Ensemble Methods for Few-Shot Classification
Dvornik, N., Mairal, J., & Schmid, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00382 · Cited by 189
Knowledge Distillation via Route Constrained Optimization
Jin, X., Peng, B., Wu, Y., Liu, Y., Liu, J., Liang, D., Yan, J., & Hu, X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00143 · Cited by 187
Self-supervised Knowledge Distillation Using Singular Value Decomposition
Lee, S., Kim, D. H., & Song, B. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_21 · Cited by 179
Data-Distortion Guided Self-Distillation for Deep Neural Networks
Xu, T.-B., & Liu, C.-L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015565 · Cited by 178
Lifelong GAN: Continual Learning for Conditional Image Generation
Zhai, M., Chen, L., Tung, F., He, J., Nawhal, M., & Mori, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00285 · Cited by 175
Distillation-Based Training for Multi-Exit Architectures
Phuong, M., & Lampert, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00144 · Cited by 175
MineGAN: Effective Knowledge Transfer From GANs to Target Domains With Few Images
Wang, Y., Gonzalez-Garcia, A., Berga, D., Herranz, L., Khan, F. S., & van de Weijer, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00935 · Cited by 171
Explaining Knowledge Distillation by Quantifying the Knowledge
Xu, C., Rao, Z., Chen, Y., & Zhang, Q. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01294 · Cited by 166
Adversarially Robust Distillation
Goldblum, M., Fowl, L., Feizi, S., & Goldstein, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5816 · Cited by 160
Knowledge Distillation from Internal Representations
Aguilar, G., Ling, Y., Zhang, Y., Yao, B., Fan, X., & Guo, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6229 · Cited by 159
MEAL: Multi-Model Ensemble via Adversarial Learning
Shen, Z., He, Z., & Xue, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014886 · Cited by 158
Self-Supervised Moving Vehicle Tracking With Stereo Sound
Gan, C., Zhao, H., Chen, P., Cox, D., & Torralba, A. B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00715 · Cited by 157
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval
Liu, Q., Xie, L., Wang, H., & Yuille, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00376 · Cited by 153
Learnable PINs: cross-modal embeddings for person identity
Nagrani, A., Albanie, S., & Zisserman, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_5 · Cited by 151
Few Sample Knowledge Distillation for Efficient Network Compression
Li, T., Li, J., Liu, Z., & Zhang, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01465 · Cited by 149
Learning Metrics From Teachers: Compact Networks for Image Embedding
Yu, L., Yazici, V. O., Liu, X., van de Weijer, J., Cheng, Y., & Ramisa, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00302 · Cited by 149
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary
Heo, B., Lee, M., Yun, S., & Choi, J. Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013771 · Cited by 149
Network of Experts for Large-Scale Image Categorization
Ahmed, K., Baig, M. H., & Torresani, L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_32 · Cited by 132
Cross-Modal Knowledge Distillation for Action Recognition
Thoker, F. M., & Gall, J. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8802909 · Cited by 128
The Knowledge Within: Methods for Data-Free Model Compression
Haroush, M., Hubara, I., Hoffer, E., & Soudry, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00852 · Cited by 125
Amalgamating Knowledge towards Comprehensive Classification
Shen, C., Wang, X., Song, J., Sun, L., & Song, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013068 · Cited by 122
Wang, L., Kim, T., & Yoon, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00834 · Cited by 120
Self-citationYang, Z., Shou, L., Gong, M., Lin, W., & Jiang, D. (2020) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3336191.3371792 · Cited by 119
SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360° Images
Lee, Y., Jeong, J., Yun, J., Cho, W., & Yoon, K. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00940 · Cited by 111
Self-citationDistilled Person Re-Identification: Towards a More Scalable System
Wu, A., Zheng, W., Guo, X., & Lai, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00128 · Cited by 110
Feature Fusion for Online Mutual Knowledge Distillation
Kim, J., Hyun, M., Chung, I., & Kwak, N. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412615 · Cited by 106 · cited 6×
“Overal insight: Is it possible to enable the student to distill knowledge by itself to achieve plausible performance? The conventional KD approaches [85], [105], [184], [216], [254] still have many setbacks to be tackled although significant performance boost has been achieved.”
BackgroundMethodInfluentialMoonshine: Distilling with Cheap Convolutions
Crowley, E. J., Gray, G., & Storkey, A. (2017) · arXiv · doi:10.48550/arxiv.1711.02613 · Cited by 106
Cross-modal adaptation for RGB-D detection
Hoffman, J., Gupta, S., Leong, J., Guadarrama, S., & Darrell, T. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487708 · Cited by 103
Relational Knowledge Transfer for Zero-Shot Learning
Wang, D., Li, Y., Lin, Y., & Zhuang, Y. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10195 · Cited by 102
Improved Knowledge Distillation via Teacher Assistant
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., & Ghasemzadeh, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5963 · Cited by 101
On Learning Associations of Faces and Voices
Kim, C., Shin, H. V., Oh, T.-H., Kaspar, A., Elgharib, M. A., & Matusik, W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20873-8_18 · Cited by 99
Preparing lessons: Improve knowledge distillation with better supervision
Wen, T., Lai, S., & Qian, X. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.102 · Cited by 99
Self-Knowledge Distillation in Natural Language Processing
Hahn, S., & Choi, H. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_050 · Cited by 98
Online Model Distillation for Efficient Video Inference
Mullapudi, R. T., Chen, S., Zhang, K., Ramanan, D., & Fatahalian, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00367 · Cited by 96
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge
Zhao, L., Peng, X., Chen, Y., Kapadia, M., & Metaxas, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00656 · Cited by 95
DualNet: Learn Complementary Features for Image Recognition
Hou, S., Liu, X., & Wang, Z. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.62 · Cited by 94
Ye, J., Ji, Y., Wang, X., Ou, K., Tao, D., & Song, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00294 · Cited by 87
Distilling Portable Generative Adversarial Networks for Image Translation
Chen, H., Wang, Y., Shu, H., Wen, C., Xu, C., Shi, B., Xu, C., & Xu, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5765 · Cited by 87
Zhang, C., & Peng, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/158 · Cited by 81
Compact Trilinear Interaction for Visual Question Answering
Do, T., Tran, H., Do, T.-T., Tjiputra, E., & Tran, Q. D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00048 · Cited by 78
Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
Shen, C., Xue, M., Wang, X., Song, J., Li, S., & Song, M. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00360 · Cited by 77
Learning to Reconstruct HDR Images from Events, with Applications to Depth and Flow Prediction
Mostafavi, M., Wang, L., & Yoon, K. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01410-2 · Cited by 73
Self-citationLiu, J., Chen, Y., & Liu, K. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016754 · Cited by 72
Self-Ensembling Attention Networks: Addressing Domain Shift for Semantic Segmentation
Xu, Y., Du, B., Zhang, L., Zhang, Q., Wang, G., & Zhang, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015581 · Cited by 72
Unifying Heterogeneous Classifiers With Distillation
Vongkulbhisal, J., Vinayavekhin, P., & Visentini-Scarzanella, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00329 · Cited by 71
Adversarial Learning of Portable Student Networks
Wang, Y., Xu, C., Xu, C., & Tao, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11667 · Cited by 70
Learning to Exploit Multiple Vision Modalities by Using Grafted Networks
Hu, Y., Delbrück, T., & Liu, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_6 · Cited by 64
Adversarial Network Compression
Belagiannis, V., Farshad, A., & Galasso, F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11018-5_37 · Cited by 63
Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning
Luo, S., Wang, X., Fang, G., Hu, Y., Tao, D., & Song, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/428 · Cited by 57
Few Shot Network Compression via Cross Distillation
Bai, H., Wu, J., King, I., & Lyu, M. R.-T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5718 · Cited by 56
ARTHuS: Adaptive Real-Time Human Segmentation in Sports Through Online Distillation
Cioppa, A., Deliège, A., Istasse, M., Vleeschouwer, C. D., & Droogenbroeck, M. V. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00306 · Cited by 55
Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN
Ye, J., Ji, Y., Wang, X., Gao, X., & Song, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01253 · Cited by 54
Deep Geometric Knowledge Distillation with Graphs
Lassance, C., Bontonou, M., Hacene, G. B., Gripon, V., Tang, J., & Ortega, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053986 · Cited by 50
GAN-Knowledge Distillation for One-Stage Object Detection
Wang, W., Hong, W., Wang, F., & Yu, J. (2020) · IEEE Access · doi:10.1109/access.2020.2983174 · Cited by 45
KNOWLEDGE TRANSFER IN DEEP CONVOLUTIONAL NEURAL NETS
Gutstein, S., Fuentes, O., & Freudenthal, E. (2008) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213008004059 · Cited by 44
Cross-Modality Distillation: A Case for Conditional Generative Adversarial Networks
Roheda, S., Riggan, B. S., Krim, H., & Dai, L. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462082 · Cited by 44
Sphere Generative Adversarial Network Based on Geometric Moment Matching
Park, S., & Kwon, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00442 · Cited by 42
Audio-Visual Model Distillation Using Acoustic Images
Pérez, A. F., Sanguineti, V., Morerio, P., & Murino, V. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093307 · Cited by 39
Deceiving Image-to-Image Translation Networks for Autonomous Driving With Adversarial Perturbations
Wang, L., Cho, W., & Yoon, K.-J. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2967289 · Cited by 38
Self-citationMulti-teacher knowledge distillation for compressed video action recognition based on deep learning
Wu, M.-C., & Chiu, C. (2019) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2019.101695 · Cited by 37
KTAN: Knowledge Transfer Adversarial Network
Liu, P., Liu, W., Ma, H., Jiang, Z., & Seok, M. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207235 · Cited by 37
Adapting Models to Signal Degradation using Distillation
Su, J.-C., & Maji, S. (2017) · British Machine Vision Conference (BMVC) · doi:10.5244/c.31.21 · Cited by 33
RGB-Depth Cross-Modal Person Re-identification
Hafner, F. M., Bhuiyan, A., Kooij, J. F. P., & Granger, É. (2019) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2019.8909838 · Cited by 32
An Adversarial Feature Distillation Method for Audio Classification
Gao, L., Mi, H., Zhu, B., Feng, D., Li, Y., & Peng, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2931656 · Cited by 31
Teacher Guided Architecture Search
Bashivan, P., Tensen, M., & DiCarlo, J. (2018) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2019.00542 · Cited by 30
Layer-Level Knowledge Distillation for Deep Neural Network Learning
Li, H., Lin, S.-C., Chen, C.-Y., & Chiang, C.-K. (2019) · Applied Sciences · doi:10.3390/app9101966 · Cited by 28
Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers
Ye, J., Wang, X., Ji, Y., Ou, K., & Song, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/573 · Cited by 28
Attention Bridging Network for Knowledge Transfer
Li, K., Zhang, Y., Li, K., Li, Y., & Fu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00530 · Cited by 28
Xu, L., Zhu, X., & Gong, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20890-5_19 · Cited by 27
Teaching Semi-Supervised Classifier via Generalized Distillation
Gong, C., Chang, X., Fang, M., & Yang, J. J. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/298 · Cited by 26
Variational Student: Learning Compact and Sparser Networks In Knowledge Distillation Framework
Hegde, S., Prasad, R., Hebbalaguppe, R., & Kumar, V. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054157 · Cited by 23
Cho, J., & Lee, M. (2019) · Sensors · doi:10.3390/s19194307 · Cited by 22
On Knowledge distillation from complex networks for response prediction
Arora, S., Khapra, M. M., & Ramaswamy, H. G. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1382 · Cited by 15
Knowledge Transfer Graph for Deep Collaborative Learning
Minami, S., Hirakawa, T., Yamashita, T., & Fujiyoshi, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69538-5_13 · Cited by 14
Knowledge Squeezed Adversarial Network CompressionPreprint
Shu, C., Li, P., Xie, Y., Qu, Y., Dai, L., & Ma, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05100 · Cited by 13
Distilling the Knowledge of Specialist Deep Neural Networks in Acoustic Scene Classification
Jung, J.-W., Heo, H.-S., Shim, H.-J., et al. (2019) · doi:10.33682/gqpj-ac63 · Cited by 13
Multistructure-Based Collaborative Online Distillation
Gao, L., Xu, L., Mi, H., Feng, D., Xu, K., & Peng, Y. (2019) · Entropy · doi:10.3390/e21040357 · Cited by 10
Pairwise Teacher-Student Network for Semi-Supervised Hashing
Zhang, S., Li, J., & Zhang, B. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00100 · Cited by 9
Heterogeneous Graph-based Knowledge Transfer for Generalized Zero-shot Learning
Wang, J., Wang, X., Jin, B., Yan, J., Zhang, W., & Hongyuan Zha, B. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412524 · Cited by 7
Teacher/Student Deep Semi-Supervised Learning for Training with Noisy Labels
Hailat, Z., & Chen, X. (2018) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2018.00147 · Cited by 6
Learning to Map Nearly Anything
Salem, T., Greenwell, C., Blanton, H., et al. (2019) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss.2019.8900646 · Cited by 5
Cited by
Lee, L.-H., Braud, T., Zhou, P., Wang, L., Xu, D., Lin, Z., Kumar, A., Bermejo, C., & Hui, P. (2021) · arXiv · doi:10.48550/arxiv.2110.05352 · Cited by 1,088
Self-citationXu, M., Ng, W. C., Lim, W. Y. B., Kang, J., Xiong, Z., Niyato, D., Yang, Q., Shen, X., & Miao, C. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3221119 · Cited by 564
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“KD is a compression method that mimics the behavior of a large DL model into a lightweight model by transferring learned knowledge [178].”
MethodStructured Pruning for Deep Convolutional Neural Networks: A Survey
He, Y., & Xiao, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3334614 · Cited by 363
Knowledge Distillation with the Reused Teacher Classifier
Chen, D., Mei, J.-P., Zhang, H., Wang, C., Feng, Y., & Chen, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01163 · Cited by 254
Metadata
- Authors
- Wang, L., & Yoon, K.-J.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-06-01
- Volume
- 44
- Issue
- 6
- Pages
- 3048-3068
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3015735225
- Cited by
- 752
- References
- 152
- FWCI
- 37.74
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.