Knowledge Distillation: A Survey
Open accessAbstract
In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher–student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.
AI snapshot
Written from the paper’s full text.
TL;DR
This paper surveys knowledge distillation (KD), covering knowledge categories, distillation schemes, teacher-student architectures, algorithms, performance comparisons, and applications. It concludes that KD is an effective and efficient model-compression technique, while identifying open challenges in theory, architecture design, and unified knowledge modeling.
Key findings
- KD systems are organized around three knowledge types—response-based, feature-based, and relation-based—and three training schemes: offline, online, and self-distillation.
- Performance comparisons on CIFAR10 and CIFAR100 show student improvements; for example, on CIFAR10 ACNs improve AlexNet from 73.24 to 78.57 (5.33↑), and on CIFAR100 TOFD improves ResNet50 from 77.42 to 84.74 (7.32↑).
- Online collaborative learning and self-knowledge distillation are reported to significantly improve deep-model performance, and KD is widely applied in visual recognition, NLP, speech recognition, recommender systems, privacy, and adversarial defense.
- The survey identifies challenges including insufficient theoretical/empirical understanding, poorly investigated teacher-student architecture design, and difficulty modeling different knowledge types in a unified complementary framework.
Methods
- Design: a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison, and applications.
- Data: reviews existing KD literature and compiles classification performance from original papers on CIFAR10 and CIFAR100, which contain 32×32 RGB images, 50,000 training images, and 10,000 testing images each, with 10 and 100 classes respectively.
- Analysis: performance tables report accuracies and improvements over individually trained teacher/student baselines; the paper states these accuracies are directly derived from the corresponding original papers.
Limitations
- Stated in the paper: theoretical and empirical understanding of KD remains insufficient, teacher-student architecture design is poorly investigated, and modeling different knowledge types in a unified complementary framework is still challenging.
- Stated in the paper: the performance comparison relies on results directly taken from original papers rather than re-running all methods under one controlled experimental setup.
- Evident from the text: the survey explicitly excludes a comprehensive review of model compression and acceleration outside knowledge distillation, and it does not describe a systematic literature-search protocol.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 262 cited sources checked · 0 supported · 0 flagged
- Findingnone given§ 5.3 Cross-Modal Distillation · p. 14
Knowledge distillation performs well in visual recognition tasks in cross-modal scenarios.
“Specifically, it can be seen that knowledge distillation performs well in visual recognition tasks in the cross-modal scenarios.”
Quote in the paperNot a substantial claim - Findingown data§ 6 Performance Comparison · p. 19
The performance of lightweight deep models (student) can be improved by knowledge transfer from high-capacity teacher models.
“- The performance of the lightweight deep models (student) can be improved by the knowledge transfer from the high-capacity teacher models.”
Quote in the paperSubstantial claim - Findingcitation§ 6 Performance Comparison · p. 19
Online knowledge distillation through collaborative learning can significantly improve the performance of deep models.
“- The online knowledge distillation through collaborative learning (Zhang et al., 2018b; Yao and Sun, 2020) can significantly improve the performance of the deep models.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Zhang et al., 2018b
Not checked in the free run.
- cites Yao and Sun, 2020
Not checked in the free run.
- Findingcitation§ 6 Performance Comparison · p. 19
Self-knowledge distillation can well improve the performance of deep models.
“- The self-knowledge distillation (Yang et al., 2019b; Yuan et al., 2020; Xu and Liu, 2019; Yun et al., 2020) can well improve the performance of the deep models.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Yang et al., 2019b
Not checked in the free run.
- cites Yuan et al., 2020
Not checked in the free run.
- cites Xu and Liu, 2019
Not checked in the free run.
- Findingown data§ 6 Performance Comparison · p. 19
Knowledge distillation can be realized on different deep models.
“- Knowledge distillation can be realized on different deep models.”
Quote in the paperNot a substantial claim
- Findingreasoning§ 6 Performance Comparison · p. 19
Model compression of different deep models can be achieved by knowledge distillation.
“- Model compression of different deep models can be achieved by knowledge distillation.”
Quote in the paperSubstantial claim - Premisecitation§ 2 Knowledge · p. 4
The parameters of the teacher model (or the connections between layers) also contain another knowledge.
“Furthermore, the parameters of the teacher model (or the connections between layers) also contain another knowledge (Liu et al., 2019c).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Liu et al., 2019c
Not checked in the free run.
- Premisecitation§ 1 Introduction · p. 2
There are not many works on either the theoretical or empirical understanding of knowledge distillation.
“Although the great success in practice, there are not too many works on either the theoretical or empirical understanding of knowledge distillation (Urner et al., 2011; Cheng et al., 2020; Phuong and Lampert, 2019a; Cho and Hariharan, 2019).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Urner et al., 2011
Not checked in the free run.
- cites Cheng et al., 2020
Not checked in the free run.
- cites Phuong and Lampert, 2019a
Not checked in the free run.
- Premisecitation§ Knowledge Distillation: A Survey · p. 1
Knowledge distillation effectively learns a small student model from a large teacher model.
“As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hinton et al., 2015
Not checked in the free run.
- Premisecitation§ 2 Knowledge · p. 4
The activations, neurons or features of intermediate layers can be used as knowledge to guide the learning of the student model.
“The activations, neurons or features of intermediate layers also can be used as the knowledge to guide the learning of the student model (Romero et al., 2015; Huang and Wang, 2017; Ahn et al., 2019; Heo et al., 2019c; Zagoruyko and Komodakis, 2017).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Romero et al., 2015
Not checked in the free run.
- cites Huang and Wang, 2017
Not checked in the free run.
- cites Zagoruyko and Komodakis, 2017
Not checked in the free run.
- Premisecitation§ 1 Introduction · p. 2
Experiments show that distillation adversely affects student learning.
“Experiments also show that distillation adversely affects the student learning.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Cho and Hariharan, 2019
Not checked in the free run.
- Premisecitation§ 1 Introduction · p. 3
Knowledge transfer from one model to another in knowledge distillation can be extended to other tasks, such as adversarial attacks, data augmentation, data privacy and security.
“Furthermore, the knowledge transfer from one model to another in knowledge distillation can be extended to other tasks, such as adversarial attacks (Papernot et al., 2016), data augmentation (Lee et al., 2019a; Gordon and Duh, 2019), data privacy and security (Wang et al., 2019a).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Papernot et al., 2016
Not checked in the free run.
- cites Lee et al., 2019a
Not checked in the free run.
- cites Wang et al., 2019a
Not checked in the free run.
- Premisecitation§ 1 Introduction · p. 3
Motivated by knowledge distillation for model compression, the idea of knowledge transfer has been applied in dataset distillation, which transfers knowledge from a large dataset into a small dataset to reduce training loads.
“Motivated by knowledge distillation for model compression, the idea of knowledge transfer has been further applied in compressing the training data, i.e., dataset distillation, which transfers the knowledge from a large dataset into a small dataset to reduce the training loads of deep models (Wang et al., 2018c; Bohdal et al., 2020).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Wang et al., 2018c
Not checked in the free run.
- cites Bohdal et al., 2020
Not checked in the free run.
- Premisecitation§ 2.1 Response-Based Knowledge · p. 5
Response-based knowledge can be used for different types of model predictions.
“The response-based knowledge can be used for different types of model predictions.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chen et al., 2017
Not checked in the free run.
- cites Zhang et al., 2019a
Not checked in the free run.
- cites Meng et al., 2019
Not checked in the free run.
- Premisecitation§ 1 Introduction · p. 2
Empirical results show that a larger model may not be a better teacher because of model capacity gap.
“Empirical results show that a larger model may not be a better teacher because of model capacity gap (Mirzadeh et al., 2020).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Mirzadeh et al., 2020
Not checked in the free run.
- Premisecitation§ 2 Knowledge · p. 4
The relationships between different activations, neurons or pairs of samples contain rich information learned by the teacher model.
“The relationships between different activations, neurons or pairs of samples contain rich information learned by the teacher model (Yim et al., 2017; Lee and Song, 2019; Liu et al., 2019g; Tung and Mori, 2019; Yu et al., 2019).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Yim et al., 2017
Not checked in the free run.
- cites Lee and Song, 2019
Not checked in the free run.
- cites Liu et al., 2019g
Not checked in the free run.
- Premisecitation§ 2.2 Feature-Based Knowledge · p. 5
Deep neural networks are good at learning multiple levels of feature representation with increasing abstraction.
“Deep neural networks are good at learning multiple levels of feature representation with increasing abstraction.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Bengio et al., 2013 source
Not checked in the free run.
- Interpretationreasoning§ 2.1 Response-Based Knowledge · p. 5
Since the soft logits are the class probability distribution, response-based knowledge distillation is limited to supervised learning.
“Since the soft logits are in fact the class probability distribution, the responsebased knowledge distillation is also limited to the supervised learning.”
Quote in the paperNot a substantial claim - Premisecitation§ 2.2 Feature-Based Knowledge · p. 5
Intermediate representations were first introduced in Fitnets to provide hints to improve the training of the student model.
“The intermediate representations were first introduced in Fitnets (Romero et al., 2015), to provide hints 1 to improve the training of the student model.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Romero et al., 2015
Not checked in the free run.
- Premisecitation§ 3.2 Online Distillation · p. 9
Some issues in offline distillation have attracted increasing attention from the research community.
“Although offline distillation methods are simple and effective, some issues in offline distillation have attracted increasing attention from the research community (Mirzadeh et al., 2020).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Mirzadeh et al., 2020
Not checked in the free run.
- Premisecitation§ 3.2 Online Distillation · p. 9
A variety of online knowledge distillation methods have been proposed, especially in the last few years.
“A variety of online knowledge distillation methods have been proposed, especially in the last few years (Zhang et al., 2018b; Chen et al., 2020a; Xie et al., 2019; Anil et al., 2018; Kim et al., 2019b; Zhou et al., 2018; Walawalkar et al., 2020; Wu and Gong, 2021; Zhang et al., 2021a).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Zhang et al., 2018b
Not checked in the free run.
- cites Chen et al., 2020a
Not checked in the free run.
- cites Xie et al., 2019
Not checked in the free run.
- Premisecitation§ 3.2 Online Distillation · p. 9
Online distillation is proposed to further improve the performance of the student model, especially when a large-capacity high performance teacher model is not available.
“To overcome the limitation of offline distillation, online distillation is proposed to further improve the performance of the student model, especially when a large-capacity high performance teacher model is not available (Zhang et al., 2018b; Chen et al., 2020a).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Zhang et al., 2018b
Not checked in the free run.
- cites Chen et al., 2020a
Not checked in the free run.
- Premisecitation§ 4 Teacher-Student Architecture · p. 11
The gap between teacher and student is further reduced by residual learning, where the assistant structure learns the residual error (Gao et al., 2021).
“The gap is further reduced by residual learning, i.e. , the assistant structure is used to learn the residual error (Gao et al., 2021).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Gao et al., 2021 source
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 10
Hahn and Choi (2019) proposed a self-knowledge distillation method in which self-knowledge consists of predicted probabilities defined by feature representations of the training model, reflecting similarities of data in feature embedding space.
“Hahn and Choi proposed a novel self-knowledge distillation method, in which the self-knowledge consists of the predicted probabilities instead of traditional soft probabilities (Hahn and Choi, 2019). These predicted probabilities are defined by the feature representations of the training model. They reflect the similarities of data in feature embedding space.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hahn and Choi, 2019
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 10
Yun et al. (2020) proposed classwise self-knowledge distillation to match output distributions of the training model between intraclass samples and augmented samples within the same source with the same model.
“Yun et al. proposed classwise self-knowledge distillation to match the output distributions of the training model between intraclass samples and augmented samples within the same source with the same model (Yun et al., 2020).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Yun et al., 2020
Not checked in the free run.
- Interpretationnone given§ 2.1 Response-Based Knowledge · p. 5
Optimizing Eq. (1) or (3) can make the logits of the student match the ones of the teacher.
“Clearly, optimizing Eq. (1) or (3) can make the logits z s of student match the ones z t of teacher.”
Quote in the paperNot a substantial claim - Premisecitation§ 4 Teacher-Student Architecture · p. 10
A variety of methods have been proposed for controlled reduction of model complexity to effectively transfer knowledge to student networks.
“To effectively transfer knowledge to student networks, a variety of methods have been proposed for a controlled reduction of the model complexity (Zhang et al., 2018b; Nowak and Corso, 2018; Crowley et al., 2018; Liu et al., 2019a,i; Wang et al., 2018a; Gu and Tresp, 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Zhang et al., 2018b
Not checked in the free run.
- cites Nowak and Corso, 2018
Not checked in the free run.
- cites Crowley et al., 2018
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 10
Self-distillation is adopted to optimize deep models with the same architecture one by one, each network distilling knowledge of the previous network using teacher-student optimization.
“Self distillation is also adopted to optimize deep models (the teacher or student networks) with the same architecture one by one (Furlanello et al., 2018; Bagherinezhad et al., 2018). Each network distills the knowledge of the previous network using a teacherstudent optimization.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Furlanello et al., 2018
Not checked in the free run.
- cites Bagherinezhad et al., 2018
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 9
Hou et al. (2019) proposed a self-attention distillation method for lane detection that uses attention maps of a network's own layers as distillation targets for its lower layers.
“Similar to the self-distillation in (Zhang et al., 2019b), a self-attention distillation method was proposed for lane detection (Hou et al., 2019). The network utilizes the attention maps of its own layers as distillation targets for its lower layers.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hou et al., 2019
Not checked in the free run.
- Premisecitation§ 2.1 Response-Based Knowledge · p. 5
Soft targets contain the informative dark knowledge from the teacher model.
“As stated in (Hinton et al., 2015), soft targets contain the informative dark knowledge from the teacher model.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hinton et al., 2015
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 9
Self-distillation's improved performance has been experimentally demonstrated in Zhang and Sabuncu (2020).
“Recently, self-distillation has been theoretically analyzed in (Mobahi et al., 2020), and its improved performance experimentally demonstrated in (Zhang and Sabuncu, 2020).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Zhang and Sabuncu, 2020
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 9
In self-distillation, the same networks are used for the teacher and the student models.
“In self-distillation, the same networks are used for the teacher and the student models (Zhang et al., 2019b; Hou et al., 2019; Zhang and Sabuncu, 2020; Yang et al., 2019b; Lee et al., 2019a; Phuong and Lampert, 2019b; Lan et al., 2018; Xu and Liu, 2019; Mobahi et al., 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Zhang et al., 2019b
Not checked in the free run.
- cites Hou et al., 2019
Not checked in the free run.
- cites Zhang and Sabuncu, 2020
Not checked in the free run.
- Premisecitation§ 4 Teacher-Student Architecture · p. 10
It is usually required to transfer knowledge from deeper and wider neural networks to shallower and thinner neural networks (Romero et al., 2015).
“It is usually required to transfer knowledge from deeper and wider neural networks to shallower and thinner neural networks (Romero et al., 2015).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Romero et al., 2015
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 10
Self-distillation and online distillation are integrated via the multiple knowledge transfer framework (Sun et al., 2021).
“For example, both self-distillation and online distillation are properly integrated via the multiple knowledge transfer framework (Sun et al., 2021).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Sun et al., 2021
Not checked in the free run.
- Interpretationreasoning§ 2.2 Feature-Based Knowledge · p. 5
Feature-based knowledge from the intermediate layers is a good extension of response-based knowledge, especially for the training of thinner and deeper networks.
“Specifically, featurebased knowledge from the intermediate layers is a good extension of response-based knowledge, especially for the training of thinner and deeper networks.”
Quote in the paperNot a substantial claim - Premisecitation§ 2.1 Response-Based Knowledge · p. 5
The effectiveness of soft targets is analogous to label smoothing or regularizers.
“From another perspective, the effectiveness of the soft targets is analogous to label smoothing (Kim and Kim, 2017) or regularizers (Muller et al., 2019; Ding et al., 2019).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Kim and Kim, 2017
Not checked in the free run.
- cites Muller et al., 2019
Not checked in the free run.
- cites Ding et al., 2019
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 9
Phuong and Lampert (2019b) proposed a distillation-based training scheme in which the early exit layer mimics the output of a later exit layer during training to reduce inference time via early exit.
“To further reduce the inference time via the early exit, Phuong and Lampert (2019b) proposed distillation-based training scheme, in which the early exit layer tries to mimic the output of later exit layer during the training.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Phuong and Lampert, 2019b
Not checked in the free run.
- Premisecitation§ 4 Teacher-Student Architecture · p. 10
The model capacity gap between a large deep neural network and a small student neural network can degrade knowledge transfer (Mirzadeh et al., 2020; Gao et al., 2021).
“The model capacity gap between the large deep neural network and a small student neural network can degrade knowledge transfer (Mirzadeh et al., 2020; Gao et al., 2021).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Mirzadeh et al., 2020
Not checked in the free run.
- cites Gao et al., 2021 source
Not checked in the free run.
- Premisecitation§ 3.3 Self-Distillation · p. 9
Zhang et al. (2019b) proposed a self-distillation method in which knowledge from deeper sections of the network is distilled into its shallow sections.
“Specifically, Zhang et al. (2019b) proposed a new self-distillation method, in which knowledge from the deeper sections of the network is distilled into its shallow sections.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Zhang et al., 2019b
Not checked in the free run.
- Premisereasoning§ 5.1 Adversarial Distillation · p. 11
Many adversarial knowledge distillation methods have been proposed to enable teacher and student networks to better understand the true data distribution.
“Inspired by this, many adversarial knowledge distillation methods have been proposed to enable the teacher and student networks to have a better understanding of the true data distribution (Wang et al., 2018e; Xu et al., 2018a; Micaelli and Storkey, 2019; Xu et al., 2018b; Liu et al., 2018; Wang et al., 2018f; Chen et al., 2019a; Shen et al., 2019d; Shu et al., 2019; Liu et al., 2020a; Belagiannis”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Wang et al., 2018e
Not checked in the free run.
- cites Xu et al., 2018a
Not checked in the free run.
- cites Micaelli and Storkey, 2019
Not checked in the free run.
- Interpretationreasoning§ 5.2 Multi-Teacher Distillation · p. 12
Multiple teacher networks can be individually and integrally used for distillation during training of a student network.
“The multiple teacher networks can be individually and integrally used for distillation during the period of training a student network.”
Quote in the paperNot a substantial claim - Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12
To transfer feature-based knowledge from multiple teachers, additional teacher branches are added to student networks to mimic intermediate features of teachers.
“To transfer feature-based knowledge from multiple teachers, additional teacher branches are added to the student networks to mimic the intermediate features of teachers (Park and Kwak, 2020; Asif et al., 2020).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Park and Kwak, 2020
Not checked in the free run.
- cites Asif et al., 2020
Not checked in the free run.
- Premisecitation§ 4 Teacher-Student Architecture · p. 11
In online setting, teacher networks are usually ensembles of student networks, in which student models share similar or the same structure with each other.
“In online setting, the teacher networks are usually ensembles of student networks, in which the student models share similar structure (or the same structure) with each other (Zhang et al., 2018b; Zhu and Gong, 2018; Furlanello et al., 2018; Chen et al., 2020a).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Zhang et al., 2018b
Not checked in the free run.
- cites Zhu and Gong, 2018
Not checked in the free run.
- cites Furlanello et al., 2018
Not checked in the free run.
- Premisecitation§ 5.1 Adversarial Distillation · p. 12
Shu et al. (2019) used squeezed knowledge as intermediate supervision to mitigate the capacity gap between teacher and student.
“Furthermore, an effective intermediate supervision, i.e. , the squeezed knowledge, was used by Shu et al. (2019) to mitigate the capacity gap between the teacher and the student.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Shu et al., 2019
Not checked in the free run.
- Interpretationreasoning§ 5.1 Adversarial Distillation · p. 12
Joint GAN and KD can generate valuable data for improving KD performance and overcoming limitations of unusable and inaccessible data.
“In summary, three main points can be concluded from the adversarial distillation methods above as follows: GAN is an effective tool to enhance the power of student learning via the teacher knowledge transfer; joint GAN and KD can generate the valuable data for improving the KD performance and overcoming the limitations of unusable and unaccessible data; KD can be used to compress GANs.”
Quote in the paperSubstantial claim - Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12
Knowledge amalgamation is designed using multiple teacher models with feature ensembles.
“Using multiple teacher models with feature ensembles, knowledge amalgamation is designed in (Shen et al., 2019a; Luo et al., 2019; Shen et al., 2019b; Luo et al., 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Shen et al., 2019a
Not checked in the free run.
- cites Luo et al., 2019
Not checked in the free run.
- cites Shen et al., 2019b
Not checked in the free run.
- Premisecitation§ 4 Teacher-Student Architecture · p. 11
The idea of dynamically searching for a knowledge transfer regime appears in knowledge distillation, e.g., automatically removing redundant layers using reinforcement learning and searching for optimal student networks given teacher networks.
“Furthermore, the idea of dynamically searching for a knowledge transfer regime also appears in knowledge distillation, e.g. , automatically removing redundant layers in a data-driven way using reinforcement learning (Ashok et al., 2018), and searching for optimal student networks given the teacher networks (Liu et al., 2019i; Xie et al., 2020; Gu and Tresp, 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Ashok et al., 2018
Not checked in the free run.
- cites Liu et al., 2019i
Not checked in the free run.
- cites Xie et al., 2020
Not checked in the free run.
- Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12
Several alternative methods simulate multiple teachers by adding different types of noise to a given teacher or by using stochastic blocks and skip connections.
“To efficiently perform knowledge transfer and explore the power of multiple teachers, several alternative methods have been proposed to simulate multiple teachers by adding different types of noise to a given teacher (Sau and Balasubramanian, 2016) or by using stochastic blocks and skip connections (Lee et al., 2019c).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Sau and Balasubramanian, 2016
Not checked in the free run.
- cites Lee et al., 2019c
Not checked in the free run.
- Interpretationreasoning§ 5.1 Adversarial Distillation · p. 12
KD can be used to compress GANs.
“In summary, three main points can be concluded from the adversarial distillation methods above as follows: GAN is an effective tool to enhance the power of student learning via the teacher knowledge transfer; joint GAN and KD can generate the valuable data for improving the KD performance and overcoming the limitations of unusable and unaccessible data; KD can be used to compress GANs.”
Quote in the paperSubstantial claim - Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12
You et al. (2017) incorporated features from intermediate layers in addition to averaged logits to encourage dissimilarity among different training samples.
“In addition to the averaged logits from all teachers, You et al. (2017) further incorporated features from the intermediate layers in order to encourage the dissimilarity among different training samples.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites You et al., 2017
Not checked in the free run.
- Premisecitation§ 5 Distillation Algorithms · p. 11
A simple yet effective idea for knowledge transfer is to directly match response-based knowledge, feature-based knowledge, or representation distributions in feature space between teacher and student.
“A simple yet very effective idea for knowledge transfer is to directly match the response-based knowledge, feature-based knowledge (Romero et al., 2015; Hinton et al., 2015) or the representation distributions in feature space (Passalis and Tefas, 2018) between the teacher model and the student model.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Romero et al., 2015
Not checked in the free run.
- cites Hinton et al., 2015
Not checked in the free run.
- cites Passalis and Tefas, 2018
Not checked in the free run.
- Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12
Born again networks address multiple teachers step-by-step, using the student at step t as teacher of the student at step t+1.
“Born again networks address multiple teachers in a step-by-step manner, i.e. , the student at the t step is used as the teacher of the student at the t +1 step (Furlanello et al., 2018), and similar ideas can be found in (Yang et al., 2019a).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Furlanello et al., 2018
Not checked in the free run.
- cites Yang et al., 2019a
Not checked in the free run.
- Premisecitation§ 5.1 Adversarial Distillation · p. 12
In the third category, adversarial knowledge distillation is carried out online, with teacher and student jointly optimized in each iteration.
“In the third category, adversarial knowledge distillation is carried out in an online manner, i.e. , the teacher and the student are jointly optimized in each iteration (Wang et al., 2018e; Chung et al., 2020), shown in Fig. 10 (c).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Wang et al., 2018e
Not checked in the free run.
- cites Chung et al., 2020
Not checked in the free run.
- Premisecitation§ 5.1 Adversarial Distillation · p. 12
Knowledge distillation can be used to compress GANs, with a learned small GAN student network mimicking a larger GAN teacher network via knowledge transfer.
“Besides, using knowledge distillation to compress GANs, a learned small GAN student network mimics a larger GAN teacher network via knowledge transfer (Aguinaldo et al., 2019; Li et al., 2020c).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Aguinaldo et al., 2019
Not checked in the free run.
- cites Li et al., 2020c
Not checked in the free run.
- Premisecitation§ 5.1 Adversarial Distillation · p. 12
In the second category, a discriminator is introduced to distinguish samples from student and teacher models by using either logits or features.
“To make student well match teacher, a discriminator in the second category is introduced to distinguish the samples from the student and the teacher models by using either the logits (Xu et al., 2018a,b) or the features (Wang et al., 2018f), shown in Fig. 10 (b).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Xu et al., 2018a
Not checked in the free run.
- cites Wang et al., 2018f
Not checked in the free run.
- Interpretationreasoning§ 5.1 Adversarial Distillation · p. 12
GAN is an effective tool to enhance the power of student learning via teacher knowledge transfer.
“In summary, three main points can be concluded from the adversarial distillation methods above as follows: GAN is an effective tool to enhance the power of student learning via the teacher knowledge transfer; joint GAN and KD can generate the valuable data for improving the KD performance and overcoming the limitations of unusable and unaccessible data; KD can be used to compress GANs.”
Quote in the paperSubstantial claim - Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12
Chen et al. (2019b) used two teacher networks, one transferring response-based knowledge and the other transferring feature-based knowledge to the student.
“To utilize both logits and intermediate features, Chen et al. (2019b) used two teacher networks, in which one teacher transfers response-based knowledge to the student and the other teacher transfers feature-based knowledge to the student.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Chen et al., 2019b
Not checked in the free run.
- Premisecitation§ 4 Teacher-Student Architecture · p. 11
The performances of small neural networks have been further improved by searching for a global structure based on efficient meta operations or blocks, inspired by NAS.
“Inspired by the success of neural architecture search (or NAS), the performances of small neural networks have been further improved by searching for a global structure based on efficient meta operations or blocks (Wu et al., 2019; Tan et al., 2019; Tan and Le, 2019; Radosavovic et al., 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Wu et al., 2019
Not checked in the free run.
- cites Tan et al., 2019
Not checked in the free run.
- cites Tan and Le, 2019
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
Tian et al. (2020) introduced a contrastive loss to transfer pair-wise relationship across different modalities.
“Tian et al. (2020) introduced a contrastive loss to transfer pair-wise relationship across different modalities.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Tian et al., 2020
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
Zhao et al. (2018) used synchronized radio signals and camera images for human pose estimation through walls or with occluded images.
“To perform human pose estimation through walls or with occluded images, Zhao et al. (2018) used synchronized radio signals and camera images.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Zhao et al., 2018
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 14
Passalis and Tefas (2018) proposed probabilistic knowledge distillation used for knowledge transfer from textual modality into visual modality.
“The probabilistic knowledge distillation proposed by Passalis and Tefas (2018) is also used for knowledge transfer from the textual modality into the visual modality.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Passalis and Tefas, 2018
Not checked in the free run.
- Interpretationcitation§ 5.4 Graph-Based Distillation · p. 15
Several methods control knowledge transfer using a graph rather than using graph-based knowledge.
“Rather than using the graph-based knowledge, several methods control knowledge transfer using a graph.”
Quote in the paperNot a substantial claim - Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
Roheda et al. (2018) proposed cross-modality distillation among missing and available modalities using GANs to improve target detection.
“To improve target detection, Roheda et al. (2018) proposed cross-modality distillation among the missing and available modelities using GANs.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Roheda et al., 2018
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
Knowledge is transferred across modalities for radio-based human pose estimation.
“Knowledge is transferred across modalities for radio-based human pose estimation.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Zhao et al., 2018
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
Thoker and Gall (2019) obtained paired samples from RGB videos and skeleton sequence to transfer knowledge learned on RGB videos to a skeleton-based human action recognition model.
“Thoker and Gall (2019) obtained paired samples from two modalities: RGB videos and skeleton sequence. The pairs are used to transfer the knowledge learned on RGB videos to a skeleton-based human action recognition model.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Thoker and Gall, 2019
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
The data or labels for some modalities might not be available during training or testing.
“The data or labels for some modalities might not be available during training or testing (Gupta et al., 2016; Garcia et al., 2018; Zhao et al., 2018; Roheda et al., 2018; Zhao et al., 2020b).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Gupta et al., 2016
Not checked in the free run.
- cites Garcia et al., 2018
Not checked in the free run.
- cites Zhao et al., 2018
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
Garcia et al. (2018) performed cross-modality distillation on depth images to generate a hallucination stream for RGB image modality to improve action recognition using only RGB images.
“To improve the action recognition performance using only RGB images, Garcia et al. (2018) performed crossmodality distillation on an additional modality, i.e. , depth image, to generate a hallucination stream for RGB image modality.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Garcia et al., 2018
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 14
Hoffman et al. (2016) proposed a modality hallucination architecture based on cross-modality distillation to improve detection performance.
“Hoffman et al. (2016) proposed a modality hallucination architecture based on cross-modality distillation to improve detection performance.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hoffman et al., 2016
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 14
Do et al. (2019) proposed a KD-based visual question answering method distilling knowledge from a trilinear interaction teacher with image-question-answer inputs into a bilinear interaction student with image-question inputs.
“Moreover, Do et al. (2019) proposed a knowledge distillation-based visual question answering method, in which knowledge from trilinear interaction teacher model with image-question-answer as inputs is distilled into the learning of a bilinear interaction student model with image-question as inputs.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Do et al., 2019
Not checked in the free run.
- Premisecitation§ 5.4 Graph-Based Distillation · p. 15
Using knowledge distillation, topological semantics of a graph convolutional teacher network are transferred as topology-aware knowledge into a graph convolutional student network (Yang et al., 2020b).
“Besides, using knowledge distillation, the topological semantics of a graph convolutional teacher network as the topology-aware knowledge are transferred into the graph convolutional student network (Yang et al., 2020b)”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Yang et al., 2020b
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 14
Cross-modal distillation methods also transfer knowledge among multiple domains.
“Besides, these cross-model distillation methods also transfer the knowledge among multiple domains (Kundu et al., 2019; Chen et al., 2019c; Su and Maji, 2017).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Kundu et al., 2019
Not checked in the free run.
- cites Chen et al., 2019c
Not checked in the free run.
- cites Su and Maji, 2017
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
Gupta et al. (2016) transferred knowledge from a teacher pretrained on one modality to a student with a new unlabeled input modality.
“Given a teacher model pretrained on one modality (e.g., RGB images) with a large number of wellannotated data samples, Gupta et al. (2016) transferred the knowledge from the teacher model to the student model with a new unlabeled input modality, such as a depth image and optical flow.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Gupta et al., 2016
Not checked in the free run.
- Premisecitation§ 5.3 Cross-Modal Distillation · p. 13
Gupta et al.'s method relies on unlabeled paired samples involving both modalities.
“Specifically, the proposed method relies on unlabeled paired samples involving both modalities, i.e. , both RGB and depth images.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Gupta et al., 2016
Not checked in the free run.
- Premisecitation§ 5.4 Graph-Based Distillation · p. 14
Park et al. (2019) directly transferred mutual relations of data samples by matching edges between a teacher graph and a student graph.
“Park et al. (2019) directly transferred the mutual relations of data samples, i.e. , to match edges between a teacher graph and a student graph.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Park et al., 2019
Not checked in the free run.
- Premisecitation§ 5.6 Data-Free Distillation · p. 15
Lopes et al. (2017) proposed a data-free KD method where transfer data to train the student is reconstructed using layer activations or layer spectral activations of the teacher network.
“In the proposed data-free knowledge distillation method (Lopes et al., 2017), the transfer data to train the student network is reconstructed by using the layer activations or layer spectral activations of the teacher network.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lopes et al., 2017
Not checked in the free run.
- Premisecitation§ 5.6 Data-Free Distillation · p. 15
A knowledge distillation method with few-shot learning is designed by distilling knowledge from a teacher model into a student neural network, with the teacher using limited labelled data.
“Similar to zero-shot learning, a knowledge distillation method with few-shot learning is designed by distilling knowledge from a teacher model into a student neural network (Kimura et al., 2018; Shen et al., 2021). The teacher uses limited labelled data.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Kimura et al., 2018
Not checked in the free run.
- cites Shen et al., 2021
Not checked in the free run.
- Interpretationnone given§ 5.8 Lifelong Distillation · p. 16
Knowledge distillation provides an effective way to preserve and transfer learned knowledge without catastrophic forgetting.
“Knowledge distillation provides an effective way to preserve and transfer learned knowledge without catastrophic forgetting.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 5.6 Data-Free Distillation · p. 16
In data-free distillation, synthesis data is usually generated from feature representations of the pre-trained teacher model.
“In summary, the synthesis data in data-free distillation is usually generated from the feature representations from the pre-trained teacher model, as shown in Fig. 14.”
Quote in the paperNot a substantial claim - Premisecitation§ 5.6 Data-Free Distillation · p. 15
Some data-free KD methods have been proposed to overcome problems with unavailable data arising from privacy, legality, security and confidentiality concerns.
“Some data-free KD methods have been proposed to overcome problems with unavailable data arising from privacy, legality, security and confidentiality concerns (Chen et al., 2019a; Lopes et al., 2017; Nayak et al., 2019; Micaelli and Storkey, 2019; Haroush et al., 2020; Ye et al., 2020; Nayak et al., 2021; Chawla et al., 2021).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chen et al., 2019a
Not checked in the free run.
- cites Lopes et al., 2017
Not checked in the free run.
- cites Nayak et al., 2019
Not checked in the free run.
- Premisecitation§ 5.8 Lifelong Distillation · p. 16
Flennerhag et al. (2019) proposed Leap, a light-weight framework for meta-learning over task manifolds by transferring knowledge from one learning process to another.
“Flennerhag et al. (2019) proposed a light-weight framework called Leap for meta-learning over task manifolds by transferring knowledge from one learning process to another.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Flennerhag et al., 2019
Not checked in the free run.
- Premisecitation§ 5.7 Quantized Distillation · p. 16
Some KD methods have been proposed using quantization in the teacher-student framework.
“Some KD methods have been proposed using the quantization process in the teacher-student framework (Polino et al., 2018; Mishra and Marr, 2018; Wei et al., 2018; Shin et al., 2019; Kim et al., 2019a).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Polino et al., 2018
Not checked in the free run.
- cites Mishra and Marr, 2018
Not checked in the free run.
- cites Wei et al., 2018
Not checked in the free run.
- Premisecitation§ 5.5 Attention-Based Distillation · p. 15
Song et al. (2018) proposed a different attentive knowledge distillation method using an attention mechanism to assign different confidence rules.
“Unlike the attention maps, a different attentive knowledge distillation method was proposed by Song et al. (2018). An attention mechanism is used to assign different confidence rules (Song et al., 2018).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Song et al., 2018
Not checked in the free run.
- Premisecitation§ 5.7 Quantized Distillation · p. 16
Polino et al. (2018) proposed a quantized distillation method to transfer knowledge to a weight-quantized student network.
“Specifically, Polino et al. (2018) proposed a quantized distillation method to transfer the knowledge to a weight-quantized student network.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Polino et al., 2018
Not checked in the free run.
- Premisecitation§ 5.6 Data-Free Distillation · p. 15
In some data-free KD methods, transfer data is generated by a GAN.
“Specifically, in (Chen et al., 2019a; Ye et al., 2020; Micaelli and Storkey, 2019; Yoo et al., 2019; Hu et al., 2020), the transfer data is generated by a GAN.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chen et al., 2019a
Not checked in the free run.
- cites Ye et al., 2020
Not checked in the free run.
- cites Micaelli and Storkey, 2019
Not checked in the free run.
- Premisecitation§ 5.6 Data-Free Distillation · p. 15
There is a new type of distillation called data distillation, similar to data-free distillation, where new training annotations of unlabeled data generated from the teacher model are employed to train a student model.
“Besides, there is a new type of distillation called data distillation, which is similar to data-free distillation (Radosavovic et al., 2018; Liu et al., 2019d; Zhang et al., 2020d). In data distillation, new training annotations of unlabeled data generated from the teacher model are employed to train a student model.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Radosavovic et al., 2018
Not checked in the free run.
- cites Liu et al., 2019d
Not checked in the free run.
- cites Zhang et al., 2020d
Not checked in the free run.
- Premisenone given§ 5.6 Data-Free Distillation · p. 15
The target data in Micaelli and Storkey (2019) and Nayak et al. (2019) is generated using information from feature representations of teacher networks.
“In fact, the target data in (Micaelli and Storkey, 2019; Nayak et al., 2019) is generated by using the information from the feature representations of teacher networks.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Micaelli and Storkey, 2019
Not checked in the free run.
- cites Nayak et al., 2019
Not checked in the free run.
- Premisecitation§ 5.8 Lifelong Distillation · p. 16
An increasing number of KD variants based on lifelong learning have been developed.
“Recently, an increasing number of KD variants, which are based on lifelong learning, have been developed (Jang et al., 2019; Flennerhag et al., 2019; Peng et al., 2019b; Liu et al., 2019e; Lee et al., 2019b; Zhai et al., 2019; Zhou et al., 2020; Shmelkov et al., 2017; Li and Hoiem, 2017; Caccia et al., 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Jang et al., 2019
Not checked in the free run.
- cites Flennerhag et al., 2019
Not checked in the free run.
- cites Peng et al., 2019b
Not checked in the free run.
- Premisecitation§ 5.8 Lifelong Distillation · p. 16
Jang et al. (2019) designed meta-transfer networks that determine what and where to transfer in the teacher-student architecture.
“Jang et al. (2019) designed meta-transfer networks that can determine what and where to transfer in the teacher-student architecture.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Jang et al., 2019
Not checked in the free run.
- Premisecitation§ 5.6 Data-Free Distillation · p. 15
Yin et al. (2020) proposed DeepInversion, which uses KD to generate synthesized images for data-free knowledge transfer.
“Yin et al. (2020) proposed DeepInversion, which uses knowledge distillation to generate synthesized images for data-free knowledge transfer.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Yin et al., 2020
Not checked in the free run.
- Interpretationcitation§ 5.5 Attention-Based Distillation · p. 15
Among attention-based KD methods, different attention transfer mechanisms are defined for distilling knowledge from teacher to student.
“Among these attention-based KD methods (Crowley et al., 2018; Huang and Wang, 2017; Srinivas and Fleuret, 2018; Zagoruyko and Komodakis, 2017), different attention transfer mechanisms are defined for distilling knowledge from the teacher network to the student network.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Crowley et al., 2018
Not checked in the free run.
- cites Huang and Wang, 2017
Not checked in the free run.
- cites Srinivas and Fleuret, 2018
Not checked in the free run.
- Premisereasoning§ 7.3 KD in Speech Recognition · p. 22
Knowledge distillation is widely studied and applied in many speech recognition tasks.
“To satisfy these requirements, knowledge distillation is widely studied and applied in many speech recognition tasks.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chebotar and Waters, 2016
Not checked in the free run.
- cites Wong and Gales, 2016
Not checked in the free run.
- cites Chan et al., 2015
Not checked in the free run.
- Rests on an assumption: More and more real-time speech recognition systems are deployed in embedded platforms with limited computational resources and fast response time.
- Rests on an assumption: The state-of-the-art deep complex models cannot satisfy the requirement of such speech recognition scenarios.
- Interpretationcitation§ 7.3 KD in Speech Recognition · p. 23
Unlike most existing traditional frame-level KD methods, sequence-level KD can perform better in some sequence models for speech recognition, such as connectionist temporal classification (CTC).
“Unlike most existing traditional frame-level KD methods, sequence-level KD can perform better in some sequence models for speech recognition, such as connectionist temporal classification (CTC) (Wong and Gales, 2016; Takashima et al., 2018; Huang et al., 2018).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Wong and Gales, 2016
Not checked in the free run.
- cites Takashima et al., 2018
Not checked in the free run.
- cites Huang et al., 2018
Not checked in the free run.
- Interpretationreasoning§ 7.3 KD in Speech Recognition · p. 23
Knowledge distillation in Asami et al. (2017) and Ghorbani et al. (2018) realizes cross-domain knowledge transfer.
“In essence, knowledge distillation in (Asami et al., 2017; Ghorbani et al., 2018) realizes the cross-domain knowledge transfer.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Asami et al., 2017
Not checked in the free run.
- cites Ghorbani et al., 2018
Not checked in the free run.
- Premisecitation§ 7.3 KD in Speech Recognition · p. 23
Knowledge distillation can be used to solve some special issues in speech recognition.
“Moreover, knowledge distillation can be used to solve some special issues in speech recognition (Bai et al., 2019; Asami et al., 2017; Ghorbani et al., 2018).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Bai et al., 2019
Not checked in the free run.
- cites Asami et al., 2017
Not checked in the free run.
- cites Ghorbani et al., 2018
Not checked in the free run.
- Premisecitation§ 7.4 KD in Other Applications · p. 23
Knowledge distillation has been successfully applied in recommender systems for deep model compression and acceleration.
“Recently, knowledge distillation has been successfully applied in recommender systems for deep model compression and acceleration (Chen et al., 2018b; Tang and Wang, 2018; Pan et al., 2019).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chen et al., 2018b
Not checked in the free run.
- cites Tang and Wang, 2018
Not checked in the free run.
- cites Pan et al., 2019 source
Not checked in the free run.
- Rests on an assumption: The full and correct leverages of external knowledge, such as in a user review or in images, play a very important role in the effectiveness of deep recommendation models.
- Rests on an assumption: Reducing the complexity and improving the efficiency of deep recommendation models is necessary.
- Interpretationreasoning§ 7.3 KD in Speech Recognition · p. 23
Many teacher-student architectures are built on RNN models because of the temporal property of speech sequences.
“- Many teacher-student architectures are built on RNN models because of the temporal property of speech sequences.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 7.3 KD in Speech Recognition · p. 23
Sequence-level knowledge distillation can be well applied to sequence models with good performance.
“- Sequence-level knowledge distillation can be well applied to sequence models with good performance.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 7.1 KD in Visual Recognition · p. 20
Knowledge distillation provides efficient and effective teacher-student learning for a variety of different visual recognition tasks, because a lightweight student network can be easily trained under the guidance of high-capacity teacher networks.
“- Knowledge distillation provides efficient and effective teacher-student learning for a variety of different visual recognition tasks, because a lightweight student network can be easily trained under the guidance of the high-capacity teacher networks.”
Quote in the paperNot a substantial claim - Premisenone given§ 8.1 Challenges · p. 24
Most KD methods leverage a combination of different kinds of knowledge, including response-based, feature-based, and relation-based knowledge.
“Most KD methods leverage a combination of different kinds of knowledge, including response-based, feature-based, and relation-based knowledge.”
Quote in the paperNot a substantial claim - Premisecitation§ 7.3 KD in Speech Recognition · p. 22
Most existing knowledge distillation methods for speech recognition use teacher-student architectures to improve the efficiency and recognition accuracy of acoustic models.
“Most existing knowledge distillation methods for speech recognition, use teacher-student architectures to improve the efficiency and recognition accuracy of acoustic models (Chan et al., 2015; Watanabe et al., 2017; Chebotar and Waters, 2016; Shen et al., 2019c; Lu et al., 2017; Shen et al., 2018, 2020; Gao et al., 2019; Shi et al., 2019c,a; Perez et al., 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chan et al., 2015
Not checked in the free run.
- cites Watanabe et al., 2017
Not checked in the free run.
- cites Chebotar and Waters, 2016
Not checked in the free run.
- Premisecitation§ 7.3 KD in Speech Recognition · p. 22
There are many knowledge distillation systems for designing lightweight deep acoustic models for speech recognition.
“There are many knowledge distillation systems for designing lightweight deep acoustic models for speech recognition (Chebotar and Waters, 2016; Wong and Gales, 2016; Chan et al., 2015; Price et al., 2016; Fukuda et al., 2017; Bai et al., 2019; Ng et al., 2018; Albanie et al., 2018; Lu et al., 2017; Shi et al.,”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chebotar and Waters, 2016
Not checked in the free run.
- cites Wong and Gales, 2016
Not checked in the free run.
- cites Chan et al., 2015
Not checked in the free run.
- Rests on an assumption: More and more real-time speech recognition systems are deployed in embedded platforms with limited computational resources and fast response time.
- Rests on an assumption: The state-of-the-art deep complex models cannot satisfy the requirement of such speech recognition scenarios.
- Premisecitation§ 8.1 Challenges · p. 24
Knowledge distillation is used as an effective strategy to solve adversarial attacks or perturbations of deep models.
“knowledge distillation is used as an effective strategy to solve adversarial attacks or perturbations of deep models (Papernot et al., 2016; Ross and Doshi-Velez, 2018; Goldblum et al., 2020; Gil et al., 2019)”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Papernot et al., 2016
Not checked in the free run.
- cites Ross and Doshi-Velez, 2018
Not checked in the free run.
- cites Goldblum et al., 2020
Not checked in the free run.
- Premisecitation§ 8.1 Challenges · p. 24
Knowledge distillation is used as an effective strategy to solve the issue of unavailable data due to privacy, confidentiality and security concerns.
“and the issue of the unavailable data due to the privacy, confidentiality and security concerns (Lopes et al., 2017; Papernot et al., 2017; Wang et al., 2019a; Bai et al., 2020; Vongkulbhisal et al., 2019).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Lopes et al., 2017
Not checked in the free run.
- cites Papernot et al., 2017
Not checked in the free run.
- cites Wang et al., 2019a
Not checked in the free run.
- Premisecitation§ 8.1 Challenges · p. 24
The perturbations of adversarial samples can be overcome by the robust outputs of the teacher networks via distillation.
“To be specific, the perturbations of the adversarial samples can be overcome by the robust outputs of the teacher networks via distillation (Ross and Doshi-Velez, 2018; Papernot et al., 2016).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Ross and Doshi-Velez, 2018
Not checked in the free run.
- cites Papernot et al., 2016
Not checked in the free run.
- Premisecitation§ 8 Conclusion and Discussion · p. 25
The assumption of linear teacher and student models enables the study of the theoretical explanations of characteristics of the student learning via distillation.
“The assumption of linear teacher and student models enables the study of the theoretical explanations of characteristics of the student learning via distillation (Phuong and Lampert, 2019a).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Phuong and Lampert, 2019a
Not checked in the free run.
- Premisecitation§ 8.1 Challenges · p. 24
To avoid exposing private data, multiple teachers access subsets of sensitive or unlabelled data and supervise the student.
“To avoid exposing the private data, multiple teachers access subsets of the sensitive or unlabelled data and supervise the student (Papernot et al., 2017; Vongkulbhisal et al., 2019).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Papernot et al., 2017
Not checked in the free run.
- cites Vongkulbhisal et al., 2019
Not checked in the free run.
- Premisecitation§ 8 Conclusion and Discussion · p. 25
Some recent works find that the student model can learn little from some teacher models due to the model capacity gap between the teacher model and the student model.
“For example, on one hand, some recent works find that the student model can learn little from some teacher models due to the model capacity gap between the teacher model and the student model (Zhang et al., 2019b; Kang et al., 2020);”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Zhang et al., 2019b
Not checked in the free run.
- cites Kang et al., 2020
Not checked in the free run.
- Premisecitation§ 8.1 Challenges · p. 24
To address privacy and security, the data to train the student network is generated by using layer activations or layer spectral activations of the teacher network via data-free distillation.
“To address the issue of privacy and security, the data to train the student network is generated by using the layer activations or layer spectral activations of the teacher network via data-free distillation (Lopes et al., 2017).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lopes et al., 2017
Not checked in the free run.
- Interpretationcitation§ 8 Conclusion and Discussion · p. 25
Apart from the knowledge and distillation algorithms, the relationship between the structures of the teacher and the student also significantly influences the performance of knowledge distillation.
“In fact, apart from the knowledge and distillation algorithms, the relationship between the structures of the teacher and the student also significantly influences the performance of knowledge distillation.”
Quote in the paperSubstantial claim - Premisecitation§ 8 Conclusion and Discussion · p. 25
From some early theoretical analysis on the capacity of neural networks, shallow networks are capable of learning the same representation as deep neural networks.
“On the other hand, from some early theoretical analysis on the capacity of neural networks, shallow networks are capable of learning the same representation as deep neural networks (Ba and Caruana, 2014).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ba and Caruana, 2014
Not checked in the free run.
- Interpretationcitation§ 8 Conclusion and Discussion · p. 25
Despite a huge number of knowledge distillation methods and applications, the understanding of knowledge distillation including theoretical explanations and empirical evaluations remains insufficient.
“Despite a huge number of the knowledge distillation methods and applications, the understanding of knowledge distillation including theoretical explanations and empirical evaluations remains insufficient (Lopez-Paz et al., 2016; Phuong and Lampert, 2019a; Cho and Hariharan, 2019).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Lopez-Paz et al., 2016
Not checked in the free run.
- cites Phuong and Lampert, 2019a
Not checked in the free run.
- cites Cho and Hariharan, 2019
Not checked in the free run.
- Classified as probably not a substantial claim. 95
- May combine two claims. 35
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given§ 7.3 KD in Speech Recognition · p. 22Better speech recognition accuracy is obtained by combining multiple acoustic modes.
“Better speech recognition accuracy is obtained by combining multiple acoustic modes.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Findingnone given§ 6 Performance Comparison · p. 19Offline and online distillation methods often transfer feature-based knowledge and response-based knowledge, respectively.
“- The offline and online distillation methods often transfer feature-based knowledge and response-based knowledge, respectively.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Premisenone given§ Knowledge Distillation: A Survey · p. 1Deep neural networks have achieved great success, but deploying them on resource-limited devices is challenging due to high computational complexity and large storage requirements.
“However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g. , mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 1 Introduction · p. 2The empirical evaluation of different forms of knowledge distillation about knowledge, distillation and mutual affection between teacher and student is not covered by Cho and Hariharan (2019).
“The empirical evaluation of different forms of knowledge distillation about knowledge, distillation and mutual affection between teacher and student is not covered by Cho and Hariharan (2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Cho and Hariharan, 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 1 Introduction · p. 2Knowledge distillation has been explored for label smoothing, for assessing the accuracy of the teacher, and for obtaining a prior for the optimal output layer geometry.
“Knowledge distillation has also been explored for label smoothing, for assessing the accuracy of the teacher and for obtaining a prior for the optimal output layer geometry (Tang et al., 2020).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Tang et al., 2020
Not checked in the free run.
May combine two claims.
Premisecitation§ 1 Introduction · p. 3Recent knowledge distillation methods have extended to teacher-student learning, mutual learning, assistant teaching, lifelong learning, and self-learning.
“Inspired by this, recent knowledge distillation methods have extended to teacher-student learning (Hinton et al., 2015), mutual learning (Zhang et al., 2018b), assistant teaching (Mirzadeh et al., 2020), lifelong learning (Zhai et al., 2019), and self-learning (Yuan et al., 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Hinton et al., 2015
Not checked in the free run.
- cites Zhang et al., 2018b
Not checked in the free run.
- cites Mirzadeh et al., 2020
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 2 Knowledge · p. 4A vanilla knowledge distillation uses the logits of a large deep model as the teacher knowledge.
“A vanilla knowledge distillation uses the logits of a large deep model as the teacher knowledge (Hinton et al., 2015; Kim et al., 2018; Ba and Caruana, 2014; Mirzadeh et al., 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Hinton et al., 2015
Not checked in the free run.
- cites Kim et al., 2018
Not checked in the free run.
- cites Ba and Caruana, 2014
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 2.1 Response-Based Knowledge · p. 5The most popular response-based knowledge for image classification is known as soft targets.
“The most popular response-based knowledge for image classification is known as soft targets (Hinton et al., 2015; Ba and Caruana, 2014).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Hinton et al., 2015
Not checked in the free run.
- cites Ba and Caruana, 2014
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based).
Premisenone given§ 2 Knowledge · p. 4In knowledge distillation, knowledge types, distillation strategies and teacher-student architectures play a crucial role in student learning.
“In knowledge distillation, knowledge types, distillation strategies and the teacher-student architectures play the crucial role in the student learning.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationreasoning§ 1 Introduction · p. 3The resulting lightweight student networks can be easily deployed in applications such as visual recognition, speech recognition, and natural language processing.
“The resulting lightweight student networks can be easily deployed in applications such as visual recognition, speech recognition, and natural language processing (NLP).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 1 Introduction · p. 2A knowledge distillation system is composed of three key components: knowledge, distillation algorithm, and teacher-student architecture.
“Basically, a knowledge distillation system is composed of three key components: knowledge, distillation algorithm, and teacher-student architecture.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisecitation§ 1 Introduction · p. 2Successful distillation relies on data geometry, optimization bias of distillation objective, and strong monotonicity of the student classifier.
“Successful distillation relies on data geometry, optimization bias of distillation objective and strong monotonicity of the student classifier.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Phuong and Lampert, 2019a
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3.3 Self-Distillation · p. 9Snapshot distillation (Yang et al., 2019b) is a variant of self-distillation in which knowledge from earlier epochs of a network is transferred into its later epochs to support supervised training within the same network.
“Snapshot distillation (Yang et al., 2019b) is a special variant of self-distillation, in which knowledge in the earlier epochs of the network (teacher) is transferred into its later epochs (student) to support a supervised training process within the same network.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Yang et al., 2019b
Not checked in the free run.
May combine two claims.
Premisecitation§ 2.3 Relation-Based Knowledge · p. 7The distilled knowledge contains not only feature information but also mutual relations of data samples.
“In fact, the distilled knowledge contains not only feature information but also mutual relations of data samples (You et al., 2017; Park et al., 2019).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites You et al., 2017
Not checked in the free run.
- cites Park et al., 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 3.1 Offline Distillation · p. 8Little attention is paid to the teacher model structure and its relationship with the student model in offline distillation.
“Little attention is paid to the teacher model structure and its relationship with the student model.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 3.3 Self-Distillation · p. 9Self-distillation can be regarded as a special case of online distillation.
“This can be regarded as a special case of online distillation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 3.1 Offline Distillation · p. 8Most previous knowledge distillation methods work offline.
“Most of previous knowledge distillation methods work offline.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3.3 Self-Distillation · p. 10Yuan et al. (2020) proposed teacher-free knowledge distillation methods based on analysis of label smoothing regularization.
“To be specific, Yuan et al. proposed teacher-free knowledge distillation methods based on the analysis of label smoothing regularization (Yuan et al., 2020).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Yuan et al., 2020
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 4 Teacher-Student Architecture · p. 11Polino et al. (2018) combined network quantization with knowledge distillation, making the student model a small and quantized version of the teacher model.
“For example, Polino et al. (2018) combined network quantization with knowledge distillation, i.e. , the student model is small and quantized version of the teacher model.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Polino et al., 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ 2.3 Relation-Based Knowledge · p. 7Traditional knowledge transfer methods often involve individual knowledge distillation, where individual soft targets of a teacher are directly distilled into the student.
“Traditional knowledge transfer methods often involve individual knowledge distillation. The individual soft targets of a teacher are directly distilled into student.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisenone given§ 3.1 Offline Distillation · p. 8In offline distillation, the first stage is usually not discussed as part of knowledge distillation; it is assumed that the teacher model is pre-defined.
“The first stage in offline distillation is usually not discussed as part of knowledge distillation, i.e. , it is assumed that the teacher model is pre-defined.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 4 Teacher-Student Architecture · p. 10Knowledge distillation was previously designed to compress an ensemble of deep neural networks (Hinton et al., 2015).
“Knowledge distillation was previously designed to compress an ensemble of deep neural networks in (Hinton et al., 2015).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hinton et al., 2015
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 2.3 Relation-Based Knowledge · p. 8Distilled knowledge can be categorized from different perspectives, such as structured knowledge of the data and privileged information about input features.
“Distilled knowledge can be categorized from different perspectives, such as structured knowledge of the data (Liu et al., 2019g; Chen et al., 2021; Peng et al., 2019a; Tung and Mori, 2019; Tian et al., 2020), privileged information about input features (Lopez-Paz et al., 2016; Vapnik and Izmailov, 2015).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Liu et al., 2019g
Not checked in the free run.
- cites Lopez-Paz et al., 2016
Not checked in the free run.
- cites Vapnik and Izmailov, 2015
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based).
Interpretationreasoning§ 4 Teacher-Student Architecture · p. 10The quality of knowledge acquisition and distillation from teacher to student is determined by how to design the teacher and student networks.
“In other words, the quality of knowledge acquisition and distillation from teacher to student is also determined by how to design the teacher and student networks.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3.3 Self-Distillation · p. 9Self-distillation has been theoretically analyzed in Mobahi et al. (2020).
“Recently, self-distillation has been theoretically analyzed in (Mobahi et al., 2020), and its improved performance experimentally demonstrated in (Zhang and Sabuncu, 2020).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Mobahi et al., 2020
Not checked in the free run.
May combine two claims.
Premisereasoning§ 2.2 Feature-Based Knowledge · p. 5Both the output of the last layer and the output of intermediate layers can be used as knowledge to supervise the training of the student model.
“Therefore, both the output of the last layer and the output of intermediate layers, i.e. , feature maps, can be used as the knowledge to supervise the training of the student model.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationcitation§ 2.1 Response-Based Knowledge · p. 5Response-based knowledge usually relies on the output of the last layer and thus fails to address the intermediate-level supervision from the teacher model, which is important for representation learning using very deep neural networks.
“However, the response-based knowledge usually relies on the output of the last layer, e.g., soft targets, and thus fails to address the intermediate-level supervision from the teacher model, which turns out to be very important for representation learning using very deep neural networks (Romero et al., 2015).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Romero et al., 2015
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3.3 Self-Distillation · p. 10Lee et al. (2019a) adopted self-distillation for data augmentation, distilling self-knowledge of augmentation into the model itself.
“In addition, the self-distillation proposed by Lee et al. (2019a) is adopted for data augmentation and the selfknowledge of augmentation is distilled into the model itself.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lee et al., 2019a
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 2.2 Feature-Based Knowledge · p. 6Feature-based knowledge transfer provides favorable information for the learning of the student model.
“Though feature-based knowledge transfer provides favorable information for the learning of the student model”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 3.2 Online Distillation · p. 9Existing online distillation methods usually fail to address the high-capacity teacher in online settings.
“However, existing online methods ( e.g. , mutual learning) usually fails to address the high-capacity teacher in online settings, making it an interesting topic to further explore the relationships between the teacher and student model in online settings.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ 3 Distillation Schemes · p. 8According to whether the teacher model is updated simultaneously with the student model or not, the learning schemes of knowledge distillation can be divided into three main categories: offline distillation, online distillation and self-distillation.
“According to whether the teacher model is updated simultaneously with the student model or not, the learning schemes of knowledge distillation can be directly divided into three main categories: offline distillation , online distillation and selfdistillation , as shown in Fig. 8.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 2.3 Relation-Based Knowledge · p. 7Relation-based knowledge further explores the relationships between different layers or data samples.
“Relation-based knowledge further explores the relationships between different layers or data samples.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 5.2 Multi-Teacher Distillation · p. 12In a typical teacher-student framework, the teacher usually has a large model or an ensemble of large models.
“In a typical teacher-student framework, the teacher usually has a large model or an ensemble of large models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12To transfer knowledge from multiple teachers, the simplest way is to use the averaged response from all teachers as the supervision signal (Hinton et al., 2015).
“To transfer knowledge from multiple teachers, the simplest way is to use the averaged response from all teachers as the supervision signal (Hinton et al., 2015).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hinton et al., 2015
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 4 Teacher-Student Architecture · p. 11Nowak and Corso (2018) proposed a structure compression method that transfers knowledge learned by multiple layers to a single layer.
“Nowak and Corso (2018) proposed a structure compression method which involves transferring the knowledge learned by multiple layers to a single layer.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Nowak and Corso, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 4 Teacher-Student Architecture · p. 11Depth-wise separable convolution has been widely used to design efficient neural networks for mobile or embedded devices.
“Recently, depth-wise separable convolution has been widely used to design efficient neural networks for mobile or embedded devices (Chollet, 2017; Howard et al., 2017; Sandler et al., 2018; Zhang et al., 2018a; Ma et al., 2018).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chollet, 2017
Not checked in the free run.
- cites Howard et al., 2017
Not checked in the free run.
- cites Sandler et al., 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given§ 5.1 Adversarial Distillation · p. 11Adversarial learning-based distillation methods using GANs can be divided into three main categories.
“As shown in Fig. 10, adversarial learning-based distillation methods, especially those methods using GANs, can be divided into three main categories as follows.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.1 Adversarial Distillation · p. 11Micaelli and Storkey (2019) utilized an adversarial generator to generate hard examples for knowledge transfer.
“Furthermore, Micaelli and Storkey (2019) utilized an adversarial generator to generate hard examples for knowledge transfer.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Micaelli and Storkey, 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 5.1 Adversarial Distillation · p. 11In a GAN, the discriminator estimates the probability that a sample comes from the training data distribution while the generator tries to fool the discriminator using generated data samples.
“Specifically, the discriminator in a GAN estimates the probability that a sample comes from the training data distribution while the generator tries to fool the discriminator using generated data samples.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone given§ 5.2 Multi-Teacher Distillation · p. 12Multiple teacher networks have turned out to be effective for training student models, usually using logits and feature representation as knowledge.
“Multiple teacher networks have turned out to be effective for training student model usually using logits and feature representation as the knowledge.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.1 Adversarial Distillation · p. 11Adversarial learning has received great attention due to its success in generative adversarial networks (Goodfellow et al., 2014).
“Recently, adversarial learning has received a great deal of attention due to its great success in generative networks, i.e. , generative adversarial networks or GANs (Goodfellow et al., 2014).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Goodfellow et al., 2014
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.1 Adversarial Distillation · p. 12Shen et al. (2019d) used multiple discriminators.
“Multiple discriminators were used by Shen et al. (2019d).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Shen et al., 2019d
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.1 Adversarial Distillation · p. 12Belagiannis et al. (2018) used unlabeled data samples to form knowledge transfer.
“Specifically, Belagiannis et al. (2018) used unlabeled data samples to form the knowledge transfer.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Belagiannis et al., 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 4 Teacher-Student Architecture · p. 11Wang et al. (2018a) progressively performed block-wise knowledge transfer from teacher networks to student networks while preserving the receptive field.
“Wang et al. (2018a) progressively performed block-wise knowledge transfer from teacher networks to student networks while preserving the receptive field.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Wang et al., 2018a
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12Fukuda et al. (2017) randomly selected one teacher from the pool of teacher networks at each iteration.
“Fukuda et al. (2017) randomly selected one teacher from the pool of teacher networks at each iteration.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Fukuda et al., 2017
Not checked in the free run.
May combine two claims.
Premisecitation§ 5.1 Adversarial Distillation · p. 11The student model has only a small capacity and cannot mimic the teacher model accurately (Mirzadeh et al., 2020).
“Simultaneously, the student model has only a small capacity and so cannot mimic the teacher model accurately (Mirzadeh et al., 2020).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Mirzadeh et al., 2020
Not checked in the free run.
May combine two claims.
Premisecitation§ 5.1 Adversarial Distillation · p. 11In the first category, an adversarial generator is trained to generate synthetic data that is either directly used as the training dataset or used to augment the training dataset.
“In the first category, an adversarial generator is trained to generate synthetic data, which is either directly used as the training dataset (Chen et al., 2019a; Ye et al., 2020) or used to augment the training dataset (Liu et al., 2018), shown in Fig. 10 (a).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chen et al., 2019a
Not checked in the free run.
- cites Ye et al., 2020
Not checked in the free run.
- cites Liu et al., 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationreasoning§ 4 Teacher-Student Architecture · p. 11To make a small student model well match a large teacher model for improving knowledge distillation performance, an adaptive teacher-student learning architecture is necessary.
“To make a small student model well match a large teacher model for improving knowledge distillation performance, the adaptive teacher-student learning architecture is necessary.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 4 Teacher-Student Architecture · p. 11Most previous works focus on designing either the structures of teacher and student models or the knowledge transfer scheme between them.
“Most previous works focus on designing either the structures of teacher and student models or the knowledge transfer scheme between them.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 15Minami et al. (2019) proposed a bidirectional graph-based diverse collaborative learning to explore diverse knowledge transfer patterns.
“Minami et al. (2019) proposed a bidirectional graph-based diverse collaborative learning to explore diverse knowledge transfer patterns.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Minami et al., 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 14In Chen et al. (2021), the graph is used to maintain the relationship between samples in high-dimensional space.
“In (Chen et al., 2021), the graph is used to maintain the relationship between samples in the high-dimensional space.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Chen et al., 2021
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 14In Zhang and Peng (2018), two graphs are constructed using logits and intermediate features to transfer knowledge from multiple self-supervised teachers to the student.
“Two graphs are then constructed using logits and intermediate features, i.e. , the logits graph and representation graph, to transfer knowledge from multiple self-supervised teachers to the student.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Zhang and Peng, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 5.2 Multi-Teacher Distillation · p. 13Multi-teacher knowledge distillation can provide rich knowledge and tailor a versatile student model because of diverse knowledge from different teachers.
“Generally, multiteacher knowledge distillation can provide rich knowledge and tailor a versatile student model because of the diverse knowledge from different teachers.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given§ 5.4 Graph-Based Distillation · p. 15Graph-based distillation can transfer informative structure knowledge of data.
“Graph-based distillation can transfer the informative structure knowledge of data.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.3 Cross-Modal Distillation · p. 13Features obtained from RGB images by the teacher are used for supervised training of the student (Gupta et al., 2016).
“The features obtained from RGB images by the teacher are then used for the supervised training of the student (Gupta et al., 2016).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Gupta et al., 2016
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 14In Liu et al. (2019g), instance features and instance relationships are modeled as vertices and edges of a graph.
“In (Liu et al., 2019g), the instance features and instance relationships are modeled as vertexes and edges of the graph, respectively.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Liu et al., 2019g
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given§ 5.3 Cross-Modal Distillation · p. 14Cross-modal knowledge transfer is challenging when there is a modality gap, e.g., lacking paired samples between different modalities.
“However, crossmodal knowledge transfer is a challenging study when there is a modality gap, e.g., lacking of the paired samples between different modalities.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisecitation§ 5.3 Cross-Modal Distillation · p. 13The idea behind paired samples is to transfer annotation or label information via pair-wise sample registration and has been widely used for cross-modal applications.
“The idea behind the paired samples is to transfer the annotation or label information via pair-wise sample registration and has been widely used for crossmodal applications (Albanie et al., 2018; Zhao et al., 2018; Thoker and Gall, 2019).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Albanie et al., 2018
Not checked in the free run.
- cites Zhao et al., 2018
Not checked in the free run.
- cites Thoker and Gall, 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 15Luo et al. (2018) considered modality discrepancy to incorporate privileged information from the source domain.
“Specifically, Luo et al. (2018) considered the modality discrepancy to incorporate privileged information from the source domain.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Luo et al., 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 15Yao et al. (2020) introduced GNNs to deal with knowledge transfer for graph-based knowledge.
“Yao et al. (2020) introduced GNNs to deal with the knowledge transfer for graph-based knowledge.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Yao et al., 2020
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 5.4 Graph-Based Distillation · p. 14Tung and Mori (2019) used a similarity matrix to represent mutual relations of activations of input pairs in teacher and student models, and the student's similarity matrix matches the teacher's.
“Tung and Mori (2019) used the similarity matrix to represent the mutual relations of the activations of the input pairs in teacher and student models. The similarity matrix of student matches that of teacher.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Tung and Mori, 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 14In Zhang and Peng (2018), each vertex represents a self-supervised teacher.
“Specifically, in (Zhang and Peng, 2018), each vertex represents a self-supervised teacher.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Zhang and Peng, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 14Lee and Song (2019) analysed intra-data relations using a multi-head graph with vertices as features from different layers in CNNs.
“Lee and Song (2019) analysed intra-data relations using a multi-head graph, in which the vertices are the features from different layers in CNNs.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lee and Song, 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 15A directed graph called a distillation graph is introduced to explore relationships between different modalities, with vertices representing modalities and edges indicating connection strength.
“A directed graph, referred to as a distillation graph is introduced to explore the relationship between different modalities. Each vertex represent a modality and the edges indicate the connection strength between one modality and another.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Luo et al., 2018
Not checked in the free run.
May combine two claims.
Premisecitation§ 5.5 Attention-Based Distillation · p. 15Attention can well reflect neuron activations of convolutional neural networks, so some attention mechanisms are used in KD to improve student performance.
“Since attention can well reflect the neuron activations of convolutional neural network, some attention mechanisms are used in knowledge distillation to improve the performance of the student network (Zagoruyko and Komodakis, 2017; Huang and Wang, 2017; Srinivas and Fleuret, 2018; Crowley et al., 2018; Song et al., 2018).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Zagoruyko and Komodakis, 2017
Not checked in the free run.
- cites Huang and Wang, 2017
Not checked in the free run.
- cites Srinivas and Fleuret, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 5.4 Graph-Based Distillation · p. 15How to properly construct a graph to model the structure knowledge of data is still a challenging study.
“However, how to properly construct graph to model the structure knowledge of data is a still challenging study.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationnone given§ 5.4 Graph-Based Distillation · p. 14The main ideas of graph-based distillation methods are to use the graph as the carrier of teacher knowledge or to use the graph to control message passing of teacher knowledge.
“The main ideas of these graph-based distillation methods are 1) to use the graph as the carrier of teacher knowledge; or 2) to use the graph to control the message passing of the teacher knowledge.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.4 Graph-Based Distillation · p. 14Knowledge transfer in Chen et al. (2021) is carried out using a locality preserving loss function.
“Knowledge transfer is then carried out using a proposed locality preserving loss function.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Chen et al., 2021
Not checked in the free run.
May combine two claims.
Premisecitation§ 5.2 Multi-Teacher Distillation · p. 12Extensions of multi-teacher distillation are used for domain adaptation via knowledge adaptation and to protect privacy and security of data.
“More interestingly, due to the special characteristics of multi-teacher distillation, its extensions are used for domain adaptation via knowledge adaptation (Ruder et al., 2017), and to protect the privacy and security of data (Vongkulbhisal et al., 2019; Papernot et al., 2017).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Ruder et al., 2017
Not checked in the free run.
- cites Vongkulbhisal et al., 2019
Not checked in the free run.
- cites Papernot et al., 2017
Not checked in the free run.
May combine two claims.
Premisecitation§ 5.4 Graph-Based Distillation · p. 14Peng et al. (2019a) matched response-based and feature-based knowledge and also used graph-based knowledge.
“Furthermore, Peng et al. (2019a) not only matched the response-based and feature-based knowledge, but also used the graph-based knowledge.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Peng et al., 2019a
Not checked in the free run.
May combine two claims.
Premisecitation§ 5.4 Graph-Based Distillation · p. 14Most knowledge distillation algorithms focus on transferring individual instance knowledge from teacher to student, while some recent methods explore intra-data relationships using graphs.
“Most of knowledge distillation algorithms focus on transferring individual instance knowledge from the teacher to the student, while some recent methods have been proposed to explore the intra-data relationships using graphs (Chen et al., 2021; Zhang and Peng, 2018; Lee and Song, 2019; Park et al., 2019; Yao et al., 2020; Ma and Mei, 2019; Hou et al., 2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Chen et al., 2021
Not checked in the free run.
- cites Zhang and Peng, 2018
Not checked in the free run.
- cites Lee and Song, 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 5.2 Multi-Teacher Distillation · p. 12Through knowledge amalgamation, many publicly available trained deep models as teachers can be reused.
“Through knowledge amalgamation, many public available trained deep models as teachers can be reused.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationreasoning§ 5.3 Cross-Modal Distillation · p. 13It is important to transfer knowledge between different modalities.
“For this reason it is important to transfer knowledge between different modalities.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.7 Quantized Distillation · p. 16Shin et al. (2019) carried out empirical analysis of deep neural networks using both distillation and quantization, considering hyperparameters such as teacher size and distillation temperature.
“Furthermore, Shin et al. (2019) carried out empirical analysis of deep neural networks using both distillation and quantization, taking into account the hyperparameters for knowledge distillation, such as the size of teacher networks and the distillation temperature.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Shin et al., 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 5.5 Attention-Based Distillation · p. 15Knowledge about feature embedding is transferred using attention map functions.
“That is to say, knowledge about feature embedding is transferred using attention map functions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ 5.7 Quantized Distillation · p. 16Knowledge distillation aims to train a small model to yield performance comparable to a complex model.
“Meanwhile, knowledge distillation aims to train a small model to yield a performance comparable to that of a complex model.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 5.7 Quantized Distillation · p. 16Wei et al. (2018) first quantized the full-precision teacher on feature maps, then transferred knowledge from quantized teacher to quantized student to ensure accurate mimicry.
“To ensure that a small student network accurately mimics a large teacher network, the full-precision teacher network is first quantized on the feature maps, and then the knowledge is transferred from the quantized teacher to a quantized student network (Wei et al., 2018).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Wei et al., 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 5.7 Quantized Distillation · p. 16Kim et al. (2019a) proposed quantization-aware knowledge distillation based on self-study of a quantized student and co-studying of teacher and student with knowledge transfer.
“Kim et al. (2019a) proposed quantization-aware knowledge distillation, which is based on self-study of a quantized student network and on the co-studying of teacher and student networks with knowledge transfer.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Kim et al., 2019a
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ 5.7 Quantized Distillation · p. 16Network quantization reduces computation complexity of neural networks by converting high-precision networks into low-precision networks.
“Network quantization reduces the computation complexity of neural networks by converting high-precision networks ( e.g. , 32-bit floating point) into low-precision networks ( e.g. , 2-bit and 8-bit).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 5.8 Lifelong Distillation · p. 16Lifelong learning accumulates previously learned knowledge and transfers learned knowledge into future learning (Chen and Liu, 2018).
“It accumulates the previously learned knowledge and also transfers the learned knowledge into future learning (Chen and Liu, 2018).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Chen and Liu, 2018
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.7 Quantized Distillation · p. 16Mishra and Marr (2018) proposed quantized KD called the 'apprentice', where a high precision teacher transfers knowledge to a small low-precision student.
“In (Mishra and Marr, 2018), the proposed quantized KD is called the 'apprentice'. A high precision teacher network transfers knowledge to a small low-precision student network.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Mishra and Marr, 2018
Not checked in the free run.
May combine two claims.
Premisenone given§ 5.8 Lifelong Distillation · p. 16The methods proposed in Jang et al. (2019), Peng et al. (2019b), Liu et al. (2019e), and Flennerhag et al. (2019) adopt meta-learning.
“The methods proposed in (Jang et al., 2019; Peng et al., 2019b; Liu et al., 2019e; Flennerhag et al., 2019) adopt meta-learning.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Jang et al., 2019
Not checked in the free run.
- cites Peng et al., 2019b
Not checked in the free run.
- cites Liu et al., 2019e
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationreasoning§ 5.6 Data-Free Distillation · p. 16Data-free distillation has shown great potential under unavailable data but remains challenging, specifically how to generate high quality diverse training data to improve model generalizability.
“Although the data-free distillation has shown a great potential under the condition of unavailable data, it remains a very challenging task, i.e., how to generate high quality diverse training data to improve the model generalizability.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisecitation§ 5.6 Data-Free Distillation · p. 15Nayak et al. (2019) proposed zero-shot knowledge distillation that does not use existing data, producing transfer data by modelling the softmax space using teacher parameters.
“Nayak et al. (2019) proposed zero-shot knowledge distillation that does not use existing data. The transfer data is produced by modelling the softmax space using the parameters of the teacher network.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Nayak et al., 2019
Not checked in the free run.
May combine two claims.
Premisecitation§ 5.7 Quantized Distillation · p. 16Boo et al. (2021) designed a self-distillation training scheme to improve performance of quantized deep models, where teacher shares model parameters of student.
“Recently, unlike the quantized distillation methods above, a self-distillation training schemes is designed to improve the performance of quantized deep models, where teacher shares model parameters of student (Boo et al., 2021).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Boo et al., 2021
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given§ 5.5 Attention-Based Distillation · p. 15The core of attention transfer is to define attention maps for feature embedding in layers of a neural network.
“The core of attention transfer is to define the attention maps for feature embedding in the layers of a neural network.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationreasoning§ 5.6 Data-Free Distillation · p. 15In data-free distillation, there is no training data; data is newly or synthetically generated.
“Just as 'data free' implies, there is no training data. Instead, the data is newly or synthetically generated.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 8.1 Challenges · p. 24Existing distillation methods can be categorized into offline distillation, online distillation and self distillation.
“Generally, the existing distillation methods can be categorized into offline distillation, online distillation and self distillation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationreasoning§ 7.1 KD in Visual Recognition · p. 20Knowledge distillation can make full use of different types of knowledge in complex data sources, such as cross-modality data, multi-domain data, multi-task data, and low-resolution data, because of flexible teacher-student architectures and knowledge transfer.
“- Knowledge distillation can make full use of the different types of knowledge in complex data sources, such as cross-modality data, multi-domain data and multi-task data and low-resolution data, because of flexible teacher-student architectures and knowledge transfer.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationreasoning§ 7.2 KD in NLP · p. 22Teacher-student knowledge transfer can easily and effectively solve many multilingual tasks, considering that knowledge from multilingual models can be transferred and shared by each other.
“- The teacher-student knowledge transfer can easily and effectively solve many multilingual tasks, considering that knowledge from multilingual models can be transferred and shared by each other.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone given§ 7.3 KD in Speech Recognition · p. 23The lightweight student model can satisfy practical requirements of speech recognition, such as real-time responses, use of limited resources and high recognition accuracy.
“- The lightweight student model can satisfy the practical requirements of speech recognition, such as realtime responses, use of limited resources and high recognition accuracy.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 7.3 KD in Speech Recognition · p. 23In general, RNN models are chosen as the teacher, which can well preserve and transfer the temporal knowledge from real acoustic data to a student model.
“In general, the RNN models are chosen as the teacher, which can well preserve and transfer the temporal knowledge from real acoustic data to a student model.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisecitation§ 8.1 Challenges · p. 24Currently, most KD methods focus on new types of knowledge or distillation loss functions, leaving the design of the teacher-student architectures poorly investigated.
“Currently, most KD methods focus on new types of knowledge or distillation loss functions, leaving the design of the teacher-student architectures poorly investigated (Nowak and Corso, 2018; Crowley et al., 2018; Kang et al., 2020; Liu et al., 2019i; Ashok et al., 2018;”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Nowak and Corso, 2018
Not checked in the free run.
- cites Crowley et al., 2018
Not checked in the free run.
- cites Kang et al., 2020
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationreasoning§ 7.2 KD in NLP · p. 22The large-capacity teacher model can transfer rich knowledge from a large number of different kinds of language data to train a small student model, so that the student can quickly complete many language tasks with effective performance.
“The large-capacity teacher model can transfer the rich knowledge from a large number of different kinds of language data to train a small student model, so that the student can quickly complete many language tasks with effective performance.”
Quote in the paperNot a substantial claim- Rests on an assumption: Conventional language models such as BERT are very time consuming and resource consuming with complex cumbersome structures.
Probably not a substantial claim: may not pass S4 (based).
Interpretationnone given§ 7.2 KD in NLP · p. 22In deep language models, sequence knowledge can be effectively transferred from large networks into small networks.
“- In deep language models, the sequence knowledge can be effectively transferred from large networks into small networks.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 7.3 KD in Speech Recognition · p. 23Sequence-level KD usually transfers feature-based knowledge from hint layers of teacher models.
“In fact, the frame-level KD always uses the responsebased knowledge, but sequence-level KD usually transfers the feature-based knowledge from hint layers of teacher models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ 8.1 Challenges · p. 24Feature-based knowledge is often used to mimic the intermediate process of the teacher.
“The featuredbased knowledge is often used to mimic the intermediate process of the teacher and the relation-based knowledge is used to capture the relationships across different samples.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 8.1 Challenges · p. 24Response-based knowledge has a similar motivation to label smoothing and model regularization.
“For example, the response-based knowledge has a similar motivation to label smoothing and the model regularization (Kim and Kim, 2017; Muller et al., 2019; Ding et al., 2019);”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Kim and Kim, 2017
Not checked in the free run.
- cites Muller et al., 2019
Not checked in the free run.
- cites Ding et al., 2019
Not checked in the free run.
Classified as probably not a substantial claim.
Study design and conductnone given§ 6 Performance Comparison · p. 18For fair comparison, the experimental classification accuracy results of the KD methods are directly derived from the corresponding original papers.
“For fair comparison, the experimental classification accuracy results (%) of the KD methods are directly derived from the corresponding original papers, as shown in Table 5 for CIFAR10 and Table 6 for CIFAR100.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 7 Applications · p. 19Knowledge distillation has been widely used in different fields of artificial intelligence, including visual recognition, speech recognition, natural language processing, and recommendation systems.
“As an effective technique for the compression and acceleration of deep neural networks, knowledge distillation has been widely used in different fields of artificial intelligence, including visual recognition, speech recognition, natural language processing (NLP), and recommendation systems.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 7.3 KD in Speech Recognition · p. 23Frame-level KD always uses response-based knowledge.
“In fact, the frame-level KD always uses the responsebased knowledge, but sequence-level KD usually transfers the feature-based knowledge from hint layers of teacher models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 7.3 KD in Speech Recognition · p. 23Knowledge distillation using teacher-student knowledge transfer can easily solve cross-domain or cross-modal speech recognition in applications such as multi-accent and multilingual speech recognition.
“- Knowledge distillation using teacher-student knowledge transfer can easily solve the cross-domain or cross-modal speech recognition in applications such as multi-accent and multilingual speech recognition.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationreasoning§ 7.2 KD in NLP · p. 22Knowledge distillation provides efficient and effective lightweight language deep models.
“- Knowledge distillation provides efficient and effective lightweight language deep models.”
Quote in the paperNot a substantial claim- Rests on an assumption: Conventional language models such as BERT are very time consuming and resource consuming with complex cumbersome structures.
May combine two claims.
Interpretationreasoning§ 6 Performance Comparison · p. 19Knowledge distillation is an effective and efficient technique of compressing deep models.
“Through the performance comparison of different knowledge distillation methods, it can be easily concluded that knowledge distillation is an effective and efficient technique of compressing deep models.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ 8.1 Challenges · p. 24Relation-based knowledge is used to capture the relationships across different samples.
“The featuredbased knowledge is often used to mimic the intermediate process of the teacher and the relation-based knowledge is used to capture the relationships across different samples.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 5.8 Lifelong Distillation · p. 16Peng et al. (2019b) designed a new knowledge transfer network architecture for few-shot image recognition that incorporates visual information from images and prior knowledge.
“Peng et al. (2019b) designed a new knowledge transfer network architecture for few-shot image recognition. The architecture simultaneously incorporates visual information from images and prior knowledge.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Peng et al., 2019b
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Premisenone given§ 8.1 Challenges · p. 24Offline distillation is usually used to transfer knowledge from a complex teacher model, while the teacher model and the student model are comparable in the settings of online distillation and self distillation.
“Offline distillation is usually used to transfer knowledge from a complex teacher model, while the teacher model and the student model are comparable in the settings of online distillation and self distillation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 8.1 Challenges · p. 24There are other special interesting applications of knowledge distillation, such as neural architecture search, interpretability of deep neural networks, and federated learning.
“Of course, there are other special interesting applications of knowledge distillation, such as neural architecture search (Macko et al., 2019; Bashivan et al., 2019), interpretability of deep neural networks (Liu et al., 2018b), and federated learning (Bistritz et al., 2020; Lin et al., 2020; Seo et al., 2020; He et al., 2020a).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Macko et al., 2019
Not checked in the free run.
- cites Bashivan et al., 2019
Not checked in the free run.
- cites Liu et al., 2018b
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone given§ 8 Conclusion and Discussion · p. 24Knowledge distillation and its applications have aroused considerable attention in recent few years.
“Knowledge distillation and its applications have aroused considerable attention in recent few years.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 8.1 Challenges · p. 24To consider the compromise between privacy and performance, Bai et al. developed a few shot network compression method via a novel layerwise knowledge distillation with few samples per class.
“To consider the compromise between privacy and performance, Bai et al. (2020) developed a few shot network compression method via a novel layerwise knowledge distillation with few samples per class.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Bai et al., 2020
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 8 Conclusion and Discussion · p. 25Some empirical evaluations and analysis on the efficacy of knowledge distillation were performed by Cho and Hariharan (2019).
“Furthermore, some empirical evaluations and analysis on the efficacy of knowledge distillation were performed by Cho and Hariharan (2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Cho and Hariharan, 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 8.1 Challenges · p. 24To protect data privacy and prevent intellectual piracy, Wang et al. proposed a private model compression framework via knowledge distillation.
“To protect data privacy and prevent intellectual piracy, Wang et al. (2019a) proposed a private model compression framework via knowledge distillation.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Wang et al., 2019a
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 8 Conclusion and Discussion · p. 25Distillation can be viewed as a form of learning with privileged information.
“For example, distillation can be viewed as a form of learning with privileged information (Lopez-Paz et al., 2016).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lopez-Paz et al., 2016
Not checked in the free run.
The paper relies on this without support.
Most extensions of knowledge distillation concentrate on compressing deep neural networks.
“Most of the extensions of knowledge distillation concentrate on compressing deep neural networks.”
The paper relies on this without support.
Offline distillation methods usually employ one-way knowledge transfer and a two-phase training procedure.
“The offline distillation methods usually employ oneway knowledge transfer and two-phase training procedure.”
The paper relies on this without support.
The complex high-capacity teacher model with huge training time cannot be avoided in offline distillation.
“However, the complex high-capacity teacher model with huge training time can not be avoided”
The paper relies on this without support.
The training of the student model in offline distillation is usually efficient under the guidance of the teacher model.
“while the training of the student model in offline distillation is usually efficient under the guidance of the teacher model.”
The paper relies on this without support.
The capacity gap between large teacher and small student always exists, and the student often largely relies on the teacher.
“Moreover, the capacity gap between large teacher and small student always exists, and student often largely relies on teacher.”
The paper relies on this without support.
Response-based knowledge distillation is simple yet effective for model compression, and has been widely used in different tasks and applications.
“The response-based knowledge distillation is simple yet effective for model compression, and has been widely used in different tasks and applications.”
The paper relies on this without support.
The main advantage of offline methods is that they are simple and easy to be implemented.
“The main advantage of offline methods is that they are simple and easy to be implemented.”
The paper relies on this without support.
The model setups of teacher and student are almost pre-fixed with unvaried sizes and structures during distillation, easily causing the model capacity gap.
“Recently, the model setups of teacher and student are almost pre-fixed with unvaried sizes and structures during distillation, so as to easily cause the model capacity gap.”
The paper relies on this without support.
It is difficult for the teacher model to perfectly learn from the true data distribution.
“In knowledge distillation, it is difficult for the teacher model to perfectly learn from the true data distribution.”
The paper relies on this without support.
Different teacher architectures can provide their own useful knowledge for a student network.
“Different teacher architectures can provide their own useful knowledge for a student network.”
The paper relies on this without support.
How to particularly design the architectures of teacher and student and why their architectures are determined by these model setups are nearly missing.
“However, how to particulary design the architectures of teacher and student and why their architectures are determined by these model setups are nearly missing.”
The paper relies on this without support.
There is a capacity gap between large teacher models and small student models that makes it difficult for the student to learn well from the teacher.
“However, there might be a capacity gap between the large teacher model and the small student model, making it difficult for the student to learn well from the teacher.”
The paper relies on this without support.
The success of knowledge transfer depends on not only the knowledge from the teacher but also the architecture of the student.
“In knowledge distillation, the success of knowledge transfer depends on not only the knowledge from the teacher but also the architecture of the student.”
The paper relies on this without support.
Conventional language models such as BERT are very time consuming and resource consuming with complex cumbersome structures.
“Conventional language models such as BERT are very time consuming and resource consuming with complex cumbersome structures.”
The paper relies on this without support.
Existing NMT models with competitive performance are very large.
“However, the existing NMT models with competitive performance is very large.”
The paper relies on this without support.
BERT is a cumbersome deep model that is not easy to be deployed.
“As a multilingual representation model, BERT has attracted attention in natural language understanding (Devlin et al., 2019), but it is also a cumbersome deep model that is not easy to be deployed.”
The paper relies on this without support.
More and more real-time speech recognition systems are deployed in embedded platforms with limited computational resources and fast response time.
“However, more and more real-time speech recognition systems are deployed in embedded platforms with limited computational resources and fast response time.”
The paper relies on this without support.
The state-of-the-art deep complex models cannot satisfy the requirement of such speech recognition scenarios.
“The stateof-the-art deep complex models cannot satisfy the requirement of such speech recognition scenarios.”
The paper relies on this without support.
Knowledge distillation in classification task is fundamental for other tasks.
“Since knowledge distillation in classification task is fundamental for other tasks, we briefly review knowledge distillation in challenging image classification settings, such as face recognition and action recognition.”
The paper relies on this without support.
The full and correct leverages of external knowledge, such as in a user review or in images, play a very important role in the effectiveness of deep recommendation models.
“The full and correct leverages of external knowledge, such as in a user review or in images, play a very important role in the effectiveness of deep recommendation models.”
The paper relies on this without support.
Reducing the complexity and improving the efficiency of deep recommendation models is necessary.
“Reducing the complexity and improving the efficiency of deep recommendation models is also very necessary.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Knowledge distillation techniques enhance deepfake detection by effectively transferring generalizable representations and decision-level knowledge from powerful teacher models to lightweight or adaptive student models.
“Knowledge distillation [252] techniques enhance deepfake detection by effectively transferring generalizable representations and decision-level knowledge from powerful teacher models to lightweight or adaptive student models.”
From Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges (2024) · cites this paper as [252]
The Knowledge section (p. 4) describes transferring teacher-model features and logits to a student, and the Performance Comparison (p. 19) says this can improve lightweight students. The passages do not establish that these techniques enhance deepfake detection; the visual-recognition discussion (p. 20) concerns face recognition instead. (LLM, conf=0.88)
Experimental — accuracy not yet measuredFine-tuning a model on step-by-step reasoning traces generated by a teacher model can transfer much of the latter's reasoning capability, substantially outperforming fine-tuning on final answers alone.
“During training, finetuning a model on step-by-step reasoning traces generated by a teacher model can transfer much of the latter's reasoning capability, substantially outperforming fine-tuning on final answers alone (Hinton et al., 2015;Gou et al., 2021;Shridhar et al., 2023;Chen et al., 2025b).”
From How to Steal Reasoning Without Reasoning Traces (2026) · cites this paper as Gou et al., 2021
The Introduction (p. 2) says a student model can mimic a teacher to achieve competitive performance, supporting the general idea of teacher-to-student knowledge transfer. The survey does not establish that training on step-by-step reasoning traces transfers reasoning capability or substantially outperforms training on final answers alone. Its reference to “step-by-step” in Multi-Teacher Distillation (p. 12) describes successive teacher–student models, not reasoning traces. (LLM, conf=0.82)
Experimental — accuracy not yet measuredModel distillation and edge computing may reduce energy consumption without sacrificing performance.
“model distillation [47] and edge computing [20], which may reduce energy consumption without sacrificing performance [119]”
From Understanding the Practices, Perceptions, and (Dis)Trust of Generative AI among Instructors: A Mixed-methods Study in the U.S. Higher Education (2025) · cites this paper as [47]
The Introduction (p. 2) describes compression into a small model without a significant drop in accuracy, and the Performance Comparison (p. 19) says distillation can produce effective lightweight models. Our copy does not say that distillation or edge computing reduces energy consumption. The separate citations [47] and [20] identify those two approaches; this source is cited for the claimed benefit. (LLM, conf=0.85)
Experimental — accuracy not yet measuredResponse-based knowledge distillation has not been explored in IOD.
“response-based method [12] has not been explored in IOD [25] yet.”
From Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation (2022) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
CIFAR100 DML improvement is overstated
MajorThe reported student accuracy of 71.10 exceeds its baseline of 68.99 by 2.11, not 8.11.
Reported 8.11 · recomputed 2.11
“| DML (Zhang et al., 2018b) | ResK | MobileNet (73.65) | ResNet32 (68.99) | 76.13, 71.10 (8.11 ↑ ) |”
Gupta et al. knowledge type differs between text and table
MinorThe text says the student is trained using features obtained by the teacher, whereas Table 4 classifies the transferred knowledge as response-based (ResK).
“The features obtained from RGB images by the teacher are then used for the supervised training of the student (Gupta et al., 2016).”
“| Gupta et al. (2016) | RGB images | Depth images | ResK | Offline |”
CIFAR100 Rocket-KD improvement does not match accuracies
MinorThe accuracy of 67.00 exceeds the student baseline of 56.30 by 10.70, rather than the reported 10.07.
Reported 10.07 · recomputed 10.7
“| Rocket-KD (Zhou et al., 2018) | FeaK | WRN-40-1 (-) | WRN-16-1 (56.30) | 67.00 (10.07 ↑ ) |”
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · see citing works
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982 · see citing works
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 4× · see citing works
“, 2018) and ShuffleNets (Zhang et al., 2018a; Ma et al., 2018); and”
MethodInfluentialDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Li, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202 · see citing works
MnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416 · see citing works
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P. D., Wu, X., Jha, S., & Swami, A. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.41 · Cited by 3,123 · see citing works
Self-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393 · see citing works
Designing Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084 · see citing works
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · see citing works
Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00454 · Cited by 1,933 · see citing works
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786 · see citing works
Relational Knowledge Distillation
Park, W., Kim, D., Lu, Y., & Cho, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00409 · Cited by 1,783 · see citing works
A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning
Yim, J., Joo, D., Bae, J., & Kim, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.754 · Cited by 1,635 · see citing works
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
Wu, B., Keutzer, K., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., & Jia, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01099 · Cited by 1,302 · see citing works
Quantized Convolutional Neural Networks for Mobile Devices
Wu, J., Leng, C., Wang, Y., Hu, Q., & Cheng, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.521 · Cited by 1,244 · see citing works
Similarity-Preserving Knowledge Distillation
Tung, F., & Mori, G. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00145 · Cited by 1,179 · see citing works
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., & Ma, K. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00381 · Cited by 1,019 · see citing works
Uninformed Students: Student-Teacher Anomaly Detection With Discriminative Latent Embeddings
Bergmann, P., Fauser, M., Sattlegger, D., & Steger, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00424 · Cited by 990 · see citing works
Sequence-Level Knowledge Distillation
Kim, Y., & Rush, A. M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1139 · Cited by 836 · see citing works
Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765695 · Cited by 817 · see citing works
Learning Lightweight Lane Detection CNNs by Self Attention Distillation
Hou, Y., Ma, Z., Liu, C., & Loy, C. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00110 · Cited by 803 · see citing works
Patient Knowledge Distillation for BERT Model Compression
Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1441 · Cited by 758 · see citing works
Structured Knowledge Distillation for Semantic Segmentation
Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., & Wang, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00271 · Cited by 711 · see citing works
Variational Information Distillation for Knowledge Transfer
Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., & Dai, Z. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00938 · Cited by 710 · see citing works
A Comprehensive Overhaul of Feature Distillation
Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., & Choi, J. Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00201 · Cited by 706 · see citing works
On the Efficacy of Knowledge Distillation
Cho, J. H., & Hariharan, B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00489 · Cited by 696 · see citing works
Through-Wall Human Pose Estimation Using Radio Signals
Zhao, M., Li, T., Alsheikh, M. A., Tian, Y., Zhao, H., Torralba, A. B., & Katabi, D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00768 · Cited by 676 · see citing works
Ross, A. S., & Doshi‐Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11504 · Cited by 621 · see citing works
Well-Read Students Learn Better: On the Importance of Pre-training Compact ModelsPreprint
Turc, I., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08962 · Cited by 620 · see citing works
Correlation Congruence for Knowledge Distillation
Peng, B., Xiao, J. J., Li, D., Zhou, S., Wu, Y., Liu, J., Zhang, Z., & Liu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00511 · Cited by 608 · see citing works
Revisiting Knowledge Distillation via Label Smoothing Regularization
Yuan, L., Tay, F. E., Li, G., Wang, T., & Feng, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00396 · Cited by 572 · see citing works
Incremental Learning of Object Detectors without Catastrophic Forgetting
Shmelkov, K., Schmid, C., & Alahari, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.368 · Cited by 559 · see citing works
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
Heo, B., Lee, M., Yun, S., & Choi, J. Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013779 · Cited by 550 · see citing works
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion
Yin, H., Molchanov, P., Alvarez, J. M., Li, Z., Mallya, A., Hoiem, D., Jha, N. K., & Kautz, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00874 · Cited by 540 · see citing works
Cross Modal Distillation for Supervision Transfer
Gupta, S., Hoffman, J., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.309 · Cited by 530 · cited 7× · see citing works
“rivacy and security of data (Vongkulbhisal et al., 2019; Papernot et al., 2017). 6.3 Cross-Modal Distillation The data or labels for some modalities might not be available during training or testing (Gupta et al., 2016; Garcia et al., 2018; Zhao et al., 2018; Roheda et al., 2018). For this reason it is important to tr…”
BackgroundMethodInfluentialLearning Deep Representations with Probabilistic Knowledge Transfer
Passalis, N., & Tefas, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_17 · Cited by 488 · see citing works
Distilling Object Detectors With Fine-Grained Feature Imitation
Wang, T., Li, Y., Zhang, X., & Feng, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00507 · Cited by 471 · see citing works
Data Distillation: Towards Omni-Supervised Learning
Radosavovic, I., Dollár, P., Girshick, R., Gkioxari, G., & He, K. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00433 · Cited by 430 · see citing works
On Compressing Deep Models by Low Rank and Sparse Decomposition
Yu, X., Liu, T., Wang, X., & Tao, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.15 · Cited by 419 · see citing works
Self-citationLearning from Multiple Teacher Networks
You, S., Xu, C., Xu, C., & Tao, D. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098135 · Cited by 413 · see citing works
Self-citationLike What You Like: Knowledge Distill via Neuron Selectivity TransferPreprint
Huang, Z., & Wang, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01219 · Cited by 401 · see citing works
Mimicking Very Efficient Network for Object Detection
Li, Q., Jin, S., & Yan, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.776 · Cited by 396 · see citing works
Data-Free Learning of Student Networks
Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu, C., Xu, C., & Tian, Q. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00361 · Cited by 372 · see citing works
Online Knowledge Distillation via Collaborative Learning
Guo, Q., Wang, X., Wu, Y., Yu, Z., Liang, D., Hu, X., & Luo, P. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01103 · Cited by 342 · see citing works
Object Relational Graph With Teacher-Recommended Learning for Video Captioning
Zhang, Z., Shi, Y., Yuan, C., Li, B., Wang, P., Hu, W., & Zha, Z.-J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01329 · Cited by 342 · see citing works
Regularizing Class-Wise Predictions via Self-Knowledge Distillation
Yun, S., Park, J., Lee, K., & Shin, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01389 · Cited by 338 · see citing works
Cross-Layer Distillation with Semantic Calibration
Chen, D., Mei, J.-P., Zhang, Y., Wang, C., Wang, Z., Feng, Y., & Chen, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16865 · Cited by 335 · see citing works
Knowledge Distillation via Instance Relationship Graph
Liu, Y., Cao, J., Li, B., Yuan, C., Hu, W., Li, Y., & Duan, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00726 · Cited by 325 · see citing works
Online Knowledge Distillation with Diverse Peers
Chen, D., Mei, J.-P., Wang, C., Feng, Y., & Chen, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5746 · Cited by 324 · see citing works
Zhang, F., Zhu, X., & Ye, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00363 · Cited by 317 · see citing works
Distilling Task-Specific Knowledge from BERT into Simple Neural NetworksPreprint
Tang, R., Lu, Y., Liu, L., Mou, L., Olga Vechtomova, A., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.12136 · Cited by 310 · see citing works
Knowledge Distillation Meets Self-supervision
Xu, G., Liu, Z., Li, X., & Loy, C. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_34 · Cited by 307 · see citing works
FastBERT: a Self-distilling BERT with Adaptive Inference Time
Liu, W., Zhou, P., Wang, Z., Zhao, Z., Deng, H., & Ju, Q. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.537 · Cited by 307 · see citing works
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency
Chen, Y., Lin, Y., Yang, M., & Huang, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00189 · Cited by 306 · see citing works
Boosting Self-Supervised Learning via Knowledge Transfer
Noroozi, M., Vinjimoor, A., Favaro, P., & Pirsiavash, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00975 · Cited by 298 · see citing works
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
Albanie, S., Nagrani, A., Vedaldi, A., & Zisserman, A. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240578 · Cited by 293 · see citing works
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Wang, W., Bao, H., Huang, S., Dong, L., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.188 · Cited by 285 · see citing works
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation
Pan, B., Cai, H., Huang, D.-A., Lee, K.-H., Gaidon, A., Adeli, E., & Niebles, J. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01088 · Cited by 281 · see citing works
Distilling Knowledge From Graph Convolutional Networks
Yang, Y., Qiu, J., Song, M., Tao, D., & Wang, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00710 · Cited by 269 · see citing works
Self-citationEfficient Knowledge Distillation from an Ensemble of Teachers
Fukuda, T., Suzuki, M., Kurata, G., et al. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-614 · Cited by 250 · see citing works
Few-Shot Image Recognition With Knowledge Transfer
Peng, Z., Li, Z., Zhang, J., Li, Y., Qi, G.-J., & Tang, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00053 · Cited by 246 · see citing works
Knowledge Adaptation for Efficient Semantic Segmentation
He, T., Shen, C., Tian, Z., Gong, D., Sun, C., & Yan, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00067 · Cited by 235 · see citing works
Adaptive multi-teacher multi-level knowledge distillation
Liu, Y., Zhang, W., & Wang, J. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.048 · Cited by 227 · see citing works
BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Clark, K., Luong, M.-T., Khandelwal, U., Manning, C. D., & Le, Q. V. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1595 · Cited by 219 · see citing works
Snapshot Distillation: Teacher-Student Optimization in One Generation
Yang, C., Xie, L., Su, C., & Yuille, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00297 · Cited by 218 · see citing works
DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer
Chen, Y., Wang, N., & Zhang, Z. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11783 · Cited by 217 · see citing works
GAN Compression: Efficient Architectures for Interactive Conditional GANs
Li, M., Lin, J., Ding, Y., Liu, Z., Zhu, J.-Y., & Han, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00533 · Cited by 216 · see citing works
Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild
Lee, K., Lee, K., Shin, J., & Lee, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00040 · Cited by 214 · see citing works
Adversarially Robust Distillation
Goldblum, M., Fowl, L., Feizi, S., & Goldstein, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5816 · Cited by 211 · see citing works
Face Model Compression by Distilling Knowledge from Neurons
Luo, P., Zhu, Z., Liu, Z., Wang, X., & Tang, X. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10449 · Cited by 210 · see citing works
Learning with Side Information through Modality Hallucination
Hoffman, J., Gupta, S., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.96 · Cited by 207 · see citing works
Modality Distillation with Multiple Stream Networks for Action Recognition
Garcia, N. C., Morerio, P., & Murino, V. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_7 · Cited by 201 · see citing works
Training Deep Neural Networks in Generations: A More Tolerant Teacher Educates Better Students
Yang, C., Xie, L., Qiao, S., & Yuille, A. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015628 · Cited by 197 · see citing works
Deep Model Compression: Distilling Knowledge from Noisy Teachers
Sau, B. B., & Balasubramanian, V. N. (2016) · arXiv.org · doi:10.48550/arxiv.1610.09650 · Cited by 196 · see citing works
Wang, T., Zhu, J.-Y., Torralba, A., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1811.10959 · Cited by 194 · see citing works
Knowledge Distillation via Route Constrained Optimization
Xiao, J. J., Peng, B., Wu, Y., et al. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00143 · Cited by 194 · see citing works
Liu, X., He, P., Chen, W., & Gao, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09482 · Cited by 188 · see citing works
Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation
Li, C., Peng, J., Yuan, L., Wang, G., Liang, X., Lin, L., & Chang, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00206 · Cited by 187 · see citing works
Lifelong GAN: Continual Learning for Conditional Image Generation
Zhai, M., Chen, L., Tung, F., He, J., Nawhal, M., & Mori, G. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00285 · Cited by 186 · see citing works
Data-Distortion Guided Self-Distillation for Deep Neural Networks
Xu, T.-B., & Liu, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015565 · Cited by 184 · see citing works
Distillation-Based Training for Multi-Exit Architectures
Phuong, M., & Lampert, C. H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00144 · Cited by 182 · see citing works
DDFlow: Learning Optical Flow with Unlabeled Data Distillation
Liu, P., King, I., Lyu, M. R.-T., & Jia, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018770 · Cited by 176 · see citing works
Distilling Knowledge from Ensembles of Neural Networks for Speech Recognition
Chebotar, Y., & Waters, A. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1190 · Cited by 174 · see citing works
Label Refinery: Improving ImageNet Classification through Label ProgressionPreprint
Bagherinezhad, H., Horton, M., Rastegari, M., & Farhadi, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.02641 · Cited by 174 · see citing works
A Multi-Task Mean Teacher for Semi-Supervised Shadow Detection
Chen, Z., Zhu, L., Wan, L., et al. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00565 · Cited by 167 · see citing works
Tang, J., & Wang, K. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220021 · Cited by 167 · see citing works
Graph Few-Shot Learning via Knowledge Transfer
Yao, H., Zhang, C., Wei, Y., Jiang, M., Wang, S., Huang, J., Chawla, N. V., & Li, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6142 · Cited by 165 · see citing works
Self-supervised Knowledge Distillation Using Singular Value Decomposition
Lee, S., Kim, D. H., & Song, B. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_21 · Cited by 162 · see citing works
Knowledge Distillation from Internal Representations
Aguilar, G., Yuan, L., Zhang, Y., Yao, B., Fan, X., & Guo, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6229 · Cited by 162 · see citing works
Heterogeneous Knowledge Distillation Using Information Flow Modeling
Passalis, N., Tzelepi, M., & Tefas, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00241 · Cited by 156 · see citing works
Few Sample Knowledge Distillation for Efficient Network Compression
Li, T., Li, J., Liu, Z., & Zhang, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01465 · Cited by 149 · see citing works
Pilzer, A., Lathuilière, S., Sebe, N., & Ricci, E. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01000 · Cited by 149 · see citing works
Inter-Region Affinity Distillation for Road Marking Segmentation
Hou, Y., Ma, Z., Liu, C., Hui, T.-W., & Loy, C. C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01250 · Cited by 145 · see citing works
Progressive Teacher-Student Learning for Early Action Prediction
Wang, X., Hu, J., Lai, J., Zhang, J., & Zheng, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00367 · Cited by 144 · see citing works
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary
Heo, B., Lee, M., Yun, S., & Choi, J. Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013771 · Cited by 141 · see citing works
Creating Something From Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing
Hu, H., Xie, L., Hong, R., et al. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00319 · Cited by 139 · see citing works
Peer Collaborative Learning for Online Knowledge Distillation
Wu, G., & Gong, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17234 · Cited by 138 · see citing works
Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning
Passalis, N., Tzelepi, M., & Tefas, A. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2995884 · Cited by 135 · see citing works
MEAL: Multi-Model Ensemble via Adversarial Learning
Shen, Z., He, Z., & Xue, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014886 · Cited by 135 · see citing works
Neural Compatibility Modeling with Attentive Knowledge Distillation
Song, X., Feng, F., Han, X., et al. (2018) · doi:10.1145/3209978.3209996 · Cited by 135 · see citing works
Reinforced Multi-Teacher Selection for Knowledge Distillation
Yuan, F., Shou, L., Pei, J., Lin, W., Gong, M., Fu, Y. H., & Jiang, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17680 · Cited by 133 · see citing works
Distilling Knowledge Learned in BERT for Text Generation
Chen, Y.-C., Gan, Z., Yu, C., Liu, J., & Liu, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.705 · Cited by 131 · see citing works
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval
Liu, Q., Xie, L., Wang, H., & Yuille, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00376 · Cited by 129 · see citing works
Explaining Knowledge Distillation by Quantifying the Knowledge
Xu, C., Rao, Z., Chen, Y., & Zhang, Q. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01294 · Cited by 126 · see citing works
Private Model Compression via Knowledge Distillation
Wang, J., Bao, W., Sun, L., Zhu, X., Cao, B., & Yu, P. S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011190 · Cited by 124 · see citing works
Distilling Knowledge From a Deep Pose Regressor Network
Saputra, M. R. U., de Gusmão, P. P. B., Almalıoğlu, Y., Markham, A., & Trigoni, N. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00035 · Cited by 124 · see citing works
Learning Metrics From Teachers: Compact Networks for Image Embedding
Yu, L., Yazici, V. O., Liu, X., van de Weijer, J., Cheng, Y., & Arnau Ramisa, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00302 · Cited by 122 · see citing works
Rocket Launching: A Universal and Efficient Framework for Training Well-Performing Light Net
Zhou, G., Fan, Y., Cui, R., Bian, W., Zhu, X., & Gai, K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11601 · Cited by 121 · see citing works
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
Passban, P., Wu, Y., Rezagholizadeh, M., & Liu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17610 · Cited by 121 · see citing works
Conditional Teacher-student Learning
Meng, Z., Li, J., Zhao, Y. H., & Gong, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683438 · Cited by 115 · see citing works
Quantization Mimic: Towards Very Tiny CNN for Object Detection
Wei, Y., Pan, X., Qin, H., Ouyang, W., & Yan, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_17 · Cited by 114 · see citing works
Improving the Interpretability of Deep Neural Networks with Knowledge Distillation
Liu, X., Wang, X., & Matwin, S. (2018) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2018.00132 · Cited by 114 · see citing works
Graph Distillation for Action Detection with Privileged Modalities
Luo, Z., Hsieh, J.-T., Jiang, L., Niebles, J. C., & Fei-Fei, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_11 · Cited by 111 · see citing works
Yan, M., Zhao, M., Xu, Z., et al. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00323 · Cited by 111 · see citing works
Distilled Person Re-Identification: Towards a More Scalable System
Wu, A., Zheng, W., Guo, X., et al. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00128 · Cited by 110 · see citing works
Feature Normalized Knowledge Distillation for Image Classification
Xu, K., Lai, R., Li, Y., & Gu, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_40 · Cited by 102 · see citing works
Improved Knowledge Distillation via Teacher Assistant
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., & Ghasemzadeh, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5963 · Cited by 101 · cited 14× · see citing works
“…including the design of knowledge (Hinton et al. 2015; Romero et al. 2015) and the loss functions for matching features or distributions matching (Huang and Wang 2017; Passalis and Tefas 2018; Zagoruyko and Komodakis 2017; Mirzadeh et al. 2020; Li et al. 2020d; Heo et al. 2019b; Asif et al. 2020).”
BackgroundMethodInfluentialA novel Enhanced Collaborative Autoencoder with knowledge distillation for top-N recommender systems
Pan, Y., He, F., & Yu, H. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.12.025 · Cited by 97 · see citing works
The Knowledge Within: Methods for Data-Free Model Compression
Haroush, M., Hubara, I., Hoffer, E., & Soudry, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00852 · Cited by 97 · see citing works
Online Model Distillation for Efficient Video Inference
Mullapudi, R. T., Chen, S., Zhang, K., Ramanan, D., & Fatahalian, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00367 · Cited by 96 · see citing works
Amalgamating Knowledge towards Comprehensive Classification
Shen, C., Wang, X., Song, J., Sun, L., & Song, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013068 · Cited by 95 · see citing works
Yang, Z., Shou, L., Gong, M., et al. (2020) · doi:10.1145/3336191.3371792 · Cited by 93 · see citing works
Cross-Modal Knowledge Distillation for Action Recognition
Thoker, F. M., & Gall, J. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8802909 · Cited by 93 · see citing works
Sequence Student-Teacher Training of Deep Neural Networks
Wong, J. H. M., & Gales, M. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-911 · Cited by 92 · see citing works
Efficient Video Classification Using Fewer Frames
Bhardwaj, S., Srinivasan, M., & Khapra, M. M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00044 · Cited by 91 · see citing works
Self-Knowledge Distillation in Natural Language Processing
Hahn, S., & Choi, H. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_050 · Cited by 90 · see citing works
Understanding and Improving Knowledge DistillationPreprint
Tang, J., Shivanna, R., Zhao, Z., Lin, D., Singh, A., Ed, H., & Jain, S. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03532 · Cited by 90 · see citing works
Packing Convolutional Neural Networks in the Frequency Domain
Wang, Y., Xu, C., Xu, C., & Tao, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2857824 · Cited by 88 · see citing works
Self-citationSiam, M., Jiang, C., Lu, S., Petrich, L., Gamal, M., Elhoseiny, M., & Jägersand, M. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794254 · Cited by 86 · see citing works
Multi-teacher Knowledge Distillation for Compressed Video Action Recognition on Deep Neural Networks
Wu, M.-C., Chiu, C., & Wu, K.-H. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682450 · Cited by 84 · see citing works
Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection
Dong, X., & Yang, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00087 · Cited by 83 · see citing works
Domain adaptation of DNN acoustic models using knowledge distillation
Asami, T., Masumura, R., Yamaguchi, Y., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953145 · Cited by 82 · see citing works
Highlight Every Step: Knowledge Distillation via Collaborative Teaching
Zhao, H., Sun, X., Dong, J., Chen, C., & Dong, Z. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3007506 · Cited by 82 · see citing works
Student-teacher network learning with enhanced features
Watanabe, S., Hori, T., Roux, J. L., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953163 · Cited by 81 · see citing works
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge
Zhao, L., Peng, X., Chen, Y., Kapadia, M., & Metaxas, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00656 · Cited by 79 · see citing works
Dynamic Kernel Distillation for Efficient Pose Estimation in Videos
Nie, X., Li, Y., Luo, L., Zhang, N., & Feng, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00704 · Cited by 79 · see citing works
Compact Trilinear Interaction for Visual Question Answering
Do, T., Tran, H., Do, T.-T., Tjiputra, E., & Tran, Q. D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00048 · Cited by 77 · see citing works
Adaptive Knowledge Distillation Based on Entropy
Kwon, K., Na, H., Lee, H., & Kim, N. S. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054698 · Cited by 76 · see citing works
Ensemble Distillation for Neural Machine TranslationPreprint
Freitag, M., Al-Onaizan, Y., & Sankaran, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01802 · Cited by 76 · see citing works
Reliable Data Distillation on Graph Convolutional Network
Zhang, W., Miao, X., Shao, Y., Jiang, J., Chen, L., Ruas, O., & Cui, B. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3389706 · Cited by 75 · see citing works
Ye, J., Ji, Y., Wang, X., Ou, K., Tao, D., & Song, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00294 · Cited by 74 · see citing works
Self-citationSearch to Distill: Pearls Are Everywhere but Not the Eyes
Liu, Y., Jia, X., Tan, M., Vemulapalli, R., Zhu, Y., Green, B., & Wang, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00756 · Cited by 73 · see citing works
Knowledge distillation for small-footprint highway networks
Lu, L., Guo, M., & Renals, S. J. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953072 · Cited by 71 · see citing works
Data-free Knowledge Distillation for Object Detection
Chawla, A., Yin, H., Molchanov, P., & Alvarez, J. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00333 · Cited by 70 · see citing works
Liu, J., Chen, Y., & Liu, K. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016754 · Cited by 70 · see citing works
Knowledge Distillation for Sequence Model
Huang, M., You, Y., Chen, Z., Qian, Y., & Yu, K. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1589 · Cited by 70 · see citing works
Distilling an ensemble of greedy dependency parsers into one MST parser
Kuncoro, A., Ballesteros, M., Kong, L., Dyer, C., & Smith, N. A. (2016) · Repositori digital de la UPF (Universitat Pompeu Fabra) · doi:10.18653/v1/d16-1180 · Cited by 67 · see citing works
Online Ensemble Model Compression Using Knowledge Distillation
Walawalkar, D., Shen, Z., & Savvides, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_2 · Cited by 66 · see citing works
Zhang, C., & Peng, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/158 · Cited by 64 · see citing works
Yue, K., Deng, J., & Zhou, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58555-6_19 · Cited by 64 · see citing works
Progressive Blockwise Knowledge Distillation for Neural Network Acceleration
Wang, H., Zhao, H., Li, X., & Tan, X. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/384 · Cited by 63 · see citing works
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition
Wang, X., Fu, T., Liao, S., Wang, S., Lei, Z., & Mei, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_20 · Cited by 62 · see citing works
Adversarial Learning of Portable Student Networks
Wang, Y., Xu, C., Xu, C., & Tao, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11667 · Cited by 61 · see citing works
Self-citationKnowledge Transfer via Dense Cross-Layer Mutual-Distillation
Yao, A., & Sun, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58555-6_18 · Cited by 61 · see citing works
Unifying Heterogeneous Classifiers With Distillation
Vongkulbhisal, J., Vinayavekhin, P., & Visentini-Scarzanella, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00329 · Cited by 60 · see citing works
Attention-Guided Answer Distillation for Machine Reading Comprehension
Hu, M., Peng, Y., Wei, F., et al. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1232 · Cited by 59 · see citing works
Adversarial Distillation for Efficient Recommendation with External Knowledge
Chen, X., Zhang, Y., Xu, H., Qin, Z., & Hongyuan Zha, B. (2018) · ACM Transactions on Information Systems · doi:10.1145/3281659 · Cited by 58 · see citing works
An Investigation of a Knowledge Distillation Method for CTC Acoustic Models
Takashima, R., Li, S., & Kawai, H. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461995 · Cited by 57 · see citing works
UM-Adapt: Unsupervised Multi-Task Adaptation Using Adversarial Cross-Task Distillation
Kundu, J. N., Lakkakula, N., & Radhakrishnan, V. B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00152 · Cited by 57 · see citing works
Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
Shen, C., Xue, M., Wang, X., et al. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00360 · Cited by 57 · see citing works
Compressing GANs using Knowledge DistillationPreprint
Aguinaldo, A., Chiang, P.-Y., Gain, A., Patil, A., Pearson, K., & Feizi, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00159 · Cited by 57 · see citing works
Fu, H., Zhou, S., Yang, Q., Tang, J., Liu, G., Liu, K., & Li, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17518 · Cited by 55 · see citing works
Spatiotemporal distilled dense-connectivity network for video action recognition
Hao, W., & Zhang, Z. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.03.005 · Cited by 55 · see citing works
Local Correlation Consistency for Knowledge Distillation
Li, X., Wu, J., Fang, H., Liao, Y., Wang, F., & Qian, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_2 · Cited by 55 · see citing works
Progressive Network Grafting for Few-Shot Knowledge Distillation
Shen, C., Wang, X., Yin, Y., Song, J., Luo, S., & Song, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16356 · Cited by 55 · see citing works
Transferring Knowledge from a RNN to a DNNPreprint
Chan, W., Ke, N. R., & Lane, I. R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.01483 · Cited by 55 · see citing works
Self-supervised Label Augmentation via Input TransformationsPreprint
Lee, H., Hwang, S. J., & Shin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05872 · Cited by 54 · see citing works
Few Shot Network Compression via Cross Distillation
Bai, H., Wu, J., King, I., & Lyu, M. R.-T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5718 · Cited by 54 · see citing works
Flexible Dataset Distillation: Learn Labels Instead of ImagesPreprint
Bohdal, O., Yang, Y., & Hospedales, T. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08572 · Cited by 54 · see citing works
Federated Knowledge DistillationPreprint
Seo, H., Park, J., Oh, S., Bennis, M., & Kim, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02367 · Cited by 53 · see citing works
TextKD-GAN: Text Generation Using Knowledge Distillation and Generative Adversarial Networks
Haidar, M. A., & Rezagholizadeh, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-18305-9_9 · Cited by 53 · see citing works
Knowledge Adaptation: Teaching to AdaptPreprint
Ruder, S., Ghaffari, P., & Breslin, J. G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02052 · Cited by 53 · see citing works
Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN
Ye, J., Ji, Y., Wang, X., Gao, X., & Song, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01253 · Cited by 52 · see citing works
Knowledge distillation across ensembles of multilingual models for low-resource languages
Cui, J., Kingsbury, B. E. D., Ramabhadran, B., Saon, G., Sercu, T., Audhkhasi, K., Sethy, A., Nußbaum-Thom, M., & Rosenberg, A. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953073 · Cited by 48 · see citing works
White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks
Gil, Y., Chai, Y., Gorodissky, O., & Berant, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1139 · Cited by 47 · see citing works
Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning
Luo, S., Wang, X., Fang, G., et al. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/428 · Cited by 47 · see citing works
Sun, P., Feng, W., Han, R., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06855 · Cited by 46 · see citing works
Towards Oracle Knowledge Distillation with Neural Architecture Search
Kang, M. S., Mun, J., & Han, B. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5866 · Cited by 46 · see citing works
Residual error based knowledge distillation
Gao, M., Wang, Y., & Wan, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.10.113 · Cited by 45 · see citing works
Low-resolution Visual Recognition via Deep Feature Distillation
Zhu, M., Han, K., Zhang, C., Lin, J., & Wang, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682926 · Cited by 44 · see citing works
QKD: Quantization-aware Knowledge Distillation
Kim, J., Bhalgat, Y., Lee, J., Patel, C., & Kwak, N. (2019) · arXiv · doi:10.48550/arxiv.1911.12491 · Cited by 44 · see citing works
Adversarial co-distillation learning for image recognition
Zhang, H., Hu, Z., Qin, W., Xu, M., & Wang, M. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107659 · Cited by 44 · see citing works
Prime-Aware Adaptive Distillation
Zhang, Y., Lan, Z., Dai, Y., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_39 · Cited by 44 · see citing works
Diverse Knowledge Distillation for End-to-End Person Search
Zhang, X., Wang, X., Bian, J.-W., Shen, C., & You, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16454 · Cited by 44 · see citing works
Differentiable Feature Aggregation Search for Knowledge Distillation
Guan, Y., Zhao, P., Wang, B., Zhang, Y., Yao, C., Bian, K., & Tang, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_28 · Cited by 42 · see citing works
Joint architecture and knowledge distillation in CNN for Chinese text recognition
Wang, Z.-R., & Du, J. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107722 · Cited by 40 · see citing works
Lai, K.-H., Zha, D., Li, Y., & Hu, X. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/435 · Cited by 40 · see citing works
Learning an Evolutionary Embedding via Massive Knowledge Distillation
Wu, X., He, R., Hu, Y., & Sun, Z. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01286-x · Cited by 36 · see citing works
Flagged first, then by citations
Research works printed but not linked: 88
- Allen-Zhu, Z., Li, Y., & Liang, Y. (2019). Learning and generalization in overparameterized neural networks, going beyond two layers. In NeurIPS.
- Anil, R., Pereyra, G., Passos, A., Ormandi, R., Dahl, G. E., & Hinton, G. E. (2018). Large scale distributed neural network training through online distillation. In ICLR.
- Arora, S., Cohen, N., & Hazan, E. (2018). On the optimization of deep networks: Implicit acceleration by overparameterization. In ICML.
- Ashok, A., Rhinehart, N., Beainy, F., & Kitani, K. M. (2018). N2N learning: Network to network compression via policy gradient reinforcement learning. In ICLR.
- Ba, J., & Caruana, R. (2014). Do deep nets really need to be deep? In NeurIPS.
- Belagiannis, V., Farshad, A., & Galasso, F. (2018). Adversarial network compression. In ECCV.
- Bistritz, I., Mann, A., & Bambos, N. (2020). Distributed Distillation for On-Device Learning. In NeurIPS.
- Brutzkus, A., & Globerson, A. (2019). Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem. In ICML.
- Caccia, M., Rodriguez, P., Ostapenko, O., Normandin, F., Lin, M., Caccia, L., Laradji, I., Rish, I., Lacoste, A., Vazquez D., & Charlin, L. (2020). Online Fast Adaptation and Knowledge Accumulation (OSAKA): a New Approach to Continual Learning. In NeurIPS.
- Chen, G., Choi, W., Yu, X., Han, T., & Chandraker, M. (2017). Learning efficient object detection models with knowledge distillation. In NeurIPS.
- Chen, H., Wang, Y., Xu, C., Xu, C., & Tao, D. (2021). Learning student networks via feature embedding. IEEE TNNLS, 32(1), 25–35.
- Chen, T., Goodfellow, I. & Shlens, J. (2016). Net2net: Accelerating learning via knowledge transfer. In ICLR.
- Chung, I., Park, S., Kim, J. & Kwak, N. (2020). Feature-map-level online adversarial knowledge distillation. In ICML.
- Courbariaux, M., Bengio, Y. & David, J. P. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS.
- Crowley, E. J., Gray, G. & Storkey, A. J. (2018). Moonshine: Distilling with cheap convolutions. In NeurIPS.
- Cui, Z., Song, T., Wang, Y., & Ji, Q. (2020). Knowledge augmented deep neural networks for joint facial expression and action unit recognition. In NeurIPS.
- Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y. & Fergus, R. (2014). Exploiting linear structure within convolutional networks for efficient evaluation. In NeurIPS.
- Dou, Q., Liu, Q., Heng, P. A., & Glocker, B. (2020). Unpaired multi-modal segmentation via knowledge distillation. IEEE TMI, 39(7), 2415–2425.
- Du, S., You, S., Li, X., Wu, J., Wang, F., Qian, C., & Zhang, C. (2020). Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. In NeurIPS.
- Fakoor, R., Mueller, J. W., Erickson, N., Chaudhari, P., & Smola, A. J. (2020). Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation. In NeurIPS.
- Flennerhag, S., Moreno, P. G., Lawrence, N. D. & Damianou, A. (2019). Transferring knowledge across learning processes. In ICLR.
- Furlanello, T., Lipton, Z., Tschannen, M., Itti, L. & Anandkumar, A. (2018). Born again neural networks. In ICML.
- Gao, Z., Chung, J., Abdelrazek, M., Leung, S., Hau, W. K., Xian, Z., et al. (2020). Privileged modality distillation for vessel border detection in intracoronary imaging. IEEE TMI, 39(5), 1524–1534.
- Ge, S., Zhao, S., Li, C., & Li, J. (2018). Low-resolution face recognition in the wild via selective knowledge distillation. IEEE TIP, 28(4), 2051–2062.
- Ge, S., Zhao, S., Li, C., Zhang, Y., & Li, J. (2020). Efficient low-resolution face recognition via bridge distillation. IEEE TIP, 29, 6898–6908.
- Gong, C., Tao, D., Liu, W., Liu, L., & Yang, J. (2017). Label propagation via teaching-to-learn and learning-to-teach. TNNLS, 28(6), 1452–1465.
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In NeurIPS.
- Han, S., Pool, J., Tran, J. & Dally, W. (2015). Learning both weights and connections for efficient neural network. In NeurIPS.
- He, C., Annavaram, M., & Avestimehr, S. (2020a). Group knowledge transfer: Federated learning of large CNNs at the edge. In NeurIPS.
- He, F., Liu, T., & Tao, D. (2020b). Why resnet works? residuals generalize. IEEE TNNLS, 31(12), 5349–5362.
- Huang, Z., Zou, Y., Bhagavatula, V., & Huang, D. (2020). Comprehensive attention self-distillation for weakly-supervised object detection. In NeurIPS.
- Jang, Y., Lee, H., Hwang, S. J. & Shin, J. (2019). Learning what and where to transfer. In ICML.
- Ji, G., & Zhu, Z. (2020). Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. In NeurIPS.
- Kim, J., Park, S. & Kwak, N. (2018). Paraphrasing complex network: Network compression via factor transfer. In NeurIPS.
- Kimura, A., Ghahramani, Z., Takeuchi, K., Iwata, T. & Ueda, N. (2018). Few-shot learning of neural networks from scratch by pseudo example optimization. In BMVC.
- Lee, K., Nguyen, L. T. & Shim, B. (2019c). Stochasticity and skip connections improve knowledge transfer. In AAAI.
- Lee, S. & Song, B. (2019). Graph-based knowledge distillation by multi-head attention network. In BMVC.
- Li, G., Zhang, J., Wang, Y., Liu, C., Tan, M., Lin, Y., Zhang, W., Feng, J., & Zhang, T. (2020b). Residual distillation: Towards portable deep neural networks without shortcuts. In NeurIPS.
- Li, J., Fu, K., Zhao, S., & Ge, S. (2019). Spatiotemporal knowledge distillation for efficient estimation of aerial video saliency. IEEE TIP, 29, 1902–1914.
- Lin, T., Kong, L., Stich, S. U., & Jaggi, M. (2020). Ensemble distillation for robust model fusion in federated learning. In NeurIPS.
- Liu, I. J., Peng, J. & Schwing, A. G. (2019a). Knowledge flow: Improve upon your teachers. In ICLR.
- Lopes, R. G., Fenu, S. & Starner, T. (2017). Data-free knowledge distillation for deep neural networks. In NeurIPS.
- Lopez-Paz, D., Bottou, L., Schölkopf, B. & Vapnik, V. (2016). Unifying distillation and privileged information. In ICLR.
- Macko, V., Weill, C., Mazzawi, H. & Gonzalvo, J. (2019). Improving neural architecture search image classifiers via ensemble learning. In NeurIPS workshop.
- Micaelli, P. & Storkey, A. J. (2019). Zero-shot knowledge transfer via adversarial belief matching. In NeurIPS.
- Mishra, A. & Marr, D. (2018). Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. In ICLR.
- Mobahi, H., Farajtabar, M., & Bartlett, P. L. (2020). Self-distillation amplifies regularization in hilbert space. In NeurIPS.
- Muller, R., Kornblith, S. & Hinton, G. E. (2019). When does label smoothing help? In NeurIPS.
- Mun, J., Lee, K., Shin, J. & Han, B. (2018). Learning to specialize with knowledge distillation for visual question answering. In NeurIPS.
- Munjal, B., Galasso, F. & Amin, S. (2019). Knowledge distillation for end-to-end person search. In BMVC.
- Nayak, G. K., Mopuri, K. R., Shaj, V., Babu, R. V. & Chakraborty, A. (2019). Zero-shot knowledge distillation in deep networks. In ICML.
- Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., et al. (2018). Parallel wavenet: Fast high-fidelity speech synthesis. In ICML.
- Papernot, N., Abadi, M., Erlingsson, U., Goodfellow, I. & Talwar, K. (2017). Semi-supervised knowledge transfer for deep learning from private training data. In ICLR.
- Peng, H., Du, H., Yu, H., Li, Q., Liao, J., & Fu, J. (2020). Cream of the crop: Distilling prioritized paths for one-shot neural architecture search. In NeurIPS.
- Phuong, M., & Lampert, C. H. (2019a). Towards understanding knowledge distillation. In ICML.
- Polino, A., Pascanu, R. & Alistarh, D. (2018). Model compression via distillation and quantization. In ICLR.
- Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). Fitnets: Hints for thin deep nets. In ICLR.
- Sindhwani, V., Sainath, T. & Kumar, S. (2015). Structured transforms for small-footprint deep learning. In NeurIPS.
- Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., & Dieleman, S. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489.
- Srinivas, S. & Fleuret, F. (2018). Knowledge transfer with jacobian matching. In ICML.
- Tan, H., Liu, X., Liu, M., Yin, B., & Li, X. (2021). KT-GAN: Knowledge-transfer generative adversarial network for text-to-image synthesis. IEEE TIP, 30, 1275–1290.
- Tan, X., Ren, Y., He, D., Qin, T., Zhao, Z. & Liu, T. Y. (2019). Multilingual neural machine translation with knowledge distillation. In ICLR.
- Tarvainen, A., & Valpola, H. (2017). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS.
- Tian, Y., Krishnan, D. & Isola, P. (2020). Contrastive representation distillation. In ICLR.
- Tu, Z., He, F., & Tao, D. (2020). Understanding generalization in recurrent neural networks. In International conference on learning representations. ICLR.
- Urban, G., Geras, K. J., Kahou, S. E., Aslan, O., Wang, S., Caruana, R., (2017). Do deep convolutional nets really need to be deep and convolutional? In ICLR.
- Vapnik, V., & Izmailov, R. (2015). Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research, 16(1), 2023–2049.
- Wang, L., & Yoon, K. J. (2020). Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. arXiv preprint arXiv:2004.05937. · arXiv
- Wang, J., Gou, L., Zhang, W., Yang, H., & Shen, H. W. (2019b). Deepvid: Deep visual interpretation and diagnosis for image classifiers via knowledge distillation. TVCG, 25(6), 2168–2180.
- Wang, M., Liu, R., Abe, N., Uchida, H., Matsunami, T., & Yamada, S. (2018b). Discover the effective strategy for face recognition model compression by improved knowledge distillation. In ICIP. · DOI
- Wang, X., Zhang, R., Sun, Y. & Qi, J. (2018e) Kdgan: Knowledge distillation with generative adversarial networks. In NeurIPS.
- Xia, S., Wang, G., Chen, Z., & Duan, Y. (2018). Complete random forest based class noise filtering learning for improving the generalizability of classifiers. IEEE TKDE, 31(11), 2063–2078.
- Xu, Z., Wu, K., Che, Z., Tang, J., & Ye, J. (2020c). Knowledge transfer in multi-task deep reinforcement learning for continuous control. In NeurIPS.
- Xu, Z., Hsu, Y. C. & Huang, J. (2018a). Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks. In ICLR workshop.
- Xu, Z., Hsu, Y. C. & Huang, J. (2018b). Training student networks for acceleration with conditional adversarial networks. In BMVC.
- Yang, J., Martinez, B., Bulat, A., & Tzimiropoulos, G. (2020a). Knowledge distillation via adaptive instance normalization. In ECCV.
- Yoo, J., Cho, M., Kim, T., & Kang, U. (2019). Knowledge extraction with no observable data. In NeurIPS.
- You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., et al. (2019). Large batch optimization for deep learning: Training bert in 76 minutes. In ICLR.
- Yuan, L., Tay, F. E., Li, G., Wang, T. & Feng, J. (2020). Revisit knowledge distillation: a teacher-free framework. In CVPR.
- Yuan, M., & Peng, Y. (2020). CKD: Cross-task knowledge distillation for text-to-image synthesis. IEEE TMM, 22(8), 1955–1968.
- Zagoruyko, S. & Komodakis, N. (2017). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR.
- Zhai, S., Cheng, Y., Zhang, Z. M. & Lu, W. (2016). Doubly convolutional neural networks. In NeurIPS.
- Zhang, L., Shi, Y., Shi, Z., Ma, K., & Bao, C. (2020a). Task-oriented feature distillation. In NeurIPS.
- Zhang, Z., & Sabuncu, M. R. (2020). Self-distillation as instance-specific label smoothing. In NeurIPS.
- Zhou C, Neubig G, Gu J (2019a) Understanding knowledge distillation in non-autoregressive machine translation. In ICLR.
- Zhou, J., Zeng, S. & Zhang, B. (2019b) Two-stage image classification supervised by a single teacher single student model. In BMVC.
- Zhou, P., Mai, L., Zhang, J., Xu, N., Wu, Z. & Davis, L. S. (2020). M2KD: Multi-model and multi-level knowledge distillation for incremental learning. In BMVC.
- Zhu, X., & Gong, S. (2018). Knowledge distillation by on-the-fly native ensemble. In NeurIPS.
Cited by
Sorted by citations
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · see citing works
“In these cases, model size may need to be reduced through techniques such as knowledge distillation, in which large-scale models teach smaller models that can be more easily deployed under practical constraints 57 .”
Federated learning on non-IID data: A survey
Zhu, H., Xu, J., Liu, S., & Jin, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.098 · Cited by 1,062 · see citing works
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113 · see citing works
“This strategy typically uses the previously-learned model as the teacher and the currently-trained model as the student, while implementing knowledge distillation (KD) [117] to mitigate catastrophic forgetting.”
MethodA review of green artificial intelligence: Towards a more sustainable future
Bolón-Canedo, V., Morán-Fernández, L., Cancela, B., & Alonso-Betanzos, A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128096 · Cited by 486 · see citing works
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023) · arXiv · doi:10.48550/arxiv.2305.13245 · Cited by 447 · see citing works
“Model distillation (Hinton et al., 2015; Gou et al., 2021) instead reduces model size at a given precision, using data generated from the larger model to finetune the smaller model.”
MethodGQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., & Sanghai, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.298 · Cited by 454 · see citing works
Knowledge Distillation with the Reused Teacher Classifier
Chen, D., Mei, J.-P., Zhang, H., Wang, C., Feng, Y., & Chen, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01163 · Cited by 254 · see citing works
Reconstructed Graph Neural Network With Knowledge Distillation for Lightweight Anomaly Detection
Zhou, X., Wu, J., Liang, W., Wang, K. I.-K., Yan, Z., Yang, L. T., & Jin, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3389714 · Cited by 145 · see citing works
Informative knowledge distillation for image anomaly segmentation
Cao, Y., Wan, Q., Shen, W., & Gao, L. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108846 · Cited by 130 · see citing works
Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation
Feng, T., Wang, M., & Yuan, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00921 · Cited by 132 · see citing works
Federated Contrastive Learning With Feature-Based Distillation for Human Activity Recognition
Xiao, Z., & Tong, H. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3510428 · Cited by 86 · see citing works
Few-Shot Class-Incremental Learning for Classification and Object Detection: A Survey
Zhang, J., Liu, L., Silvén, O., Pietikäinen, M., & Hu, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3529038 · Cited by 84 · see citing works
“2) Knowledge Distillation-Based Methods: In continuous learning, KD is widely employed to transfer knowledge from an old model, known as the “teacher model,” to a new model, referred to as the “student model” [91].”
Lyu, W., Zhang, S., Chung, T., Sun, Y., & Zhang, Y. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100383 · Cited by 68 · see citing works
Yuan, H., Yu, K., Xie, F., Liu, M., & Sun, S. (2024) · Medicine Advances · doi:10.1002/med4.75 · Cited by 61 · see citing works
Distilled GPT for source code summarization
Su, C.-Y., & McMillan, C. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00421-4 · Cited by 49 · see citing works
Knowledge from the Original Network: Restore a Better Pruned Network with Knowledge Distillation
Chen, L., Chen, Y., Xi, J., & Le, X. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-020-00248-y · Cited by 48 · see citing works
Discovering cognitive strategies with tiny recurrent neural networks
Ji-An, L., Benna, M. K., & Mattar, M. G. (2025) · Nature · doi:10.1038/s41586-025-09142-4 · Cited by 64 · see citing works
Liu, C.-Y., Kuo, E.-J., Abraham Lin, C.-H., Gemsun Young, J., Chang, Y.-J., Hsieh, M.-H., & Goan, H.-S. (2025) · Quantum Machine Intelligence · doi:10.1007/s42484-025-00305-0 · Cited by 40 · see citing works
Africa, ChatGPT, and Generative AI Systems: Ethical Benefits, Concerns, and the Need for Governance
Wakunuma, K., & Eke, D. (2024) · Philosophies · doi:10.3390/philosophies9030080 · Cited by 33 · see citing works
ModelShield: Adaptive and Robust Watermark Against Model Extraction Attack
Pang, K., Qi, T., Wu, C., Bai, M., Jiang, M., & Huang, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3530691 · Cited by 29 · see citing works
Lamaakal, I., Yahyati, C., Maleh, Y., El Makkaoui, K., Ouahbi, I., El-Latif, A. A. A., Zomorodi, M., & El-Rahiem, B. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-26297-2 · Cited by 26 · see citing works
“In the proposed XTinyHAR framework, we adopt an offline knowledge distillation 45 paradigm where the multimodal teacher is pre-trained and fixed prior to student training 46 .”
Wingate, D., Shoeybi, M., & Sorensen, T. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.412 · Cited by 26 · see citing works
From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26 · see citing works
Ghani, B., Kalkman, V. J., Planqué, B., Vellinga, W.-P., Gill, L., & Stowell, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-00996-2 · Cited by 22 · see citing works
Lifelong Learning of Large Language Model Based Agents: A Roadmap
Zheng, J., Shi, C., Cai, X., Li, Q., Zhang, D., Li, C., Yu, D., & Ma, Q. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3650546 · Cited by 7 · see citing works
“Additionally, by combining knowledge distillation [63], [64], [65] and regularization methods, some studies [62], [66], [67], [68] leverage the high-performance capability of the teacher model in one modality to transfer its knowledge to the student model through constraint-based mapping.”
Generative AI-based data augmentation for improved bioacoustic classification in noisy environments
Gibbons, A., King, E., Donohue, I., & Parnell, A. (2024) · arXiv · doi:10.48550/arxiv.2412.01530 · Cited by 6 · see citing works
Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges
Liu, P., Tao, Q., & Zhou, J. T. (2024) · arXiv · doi:10.48550/arxiv.2406.06965 · Cited by 6 · see citing works
Liu, P., Tao, Q., & Zhou, J. (2024) · Proceedings of the 1st ACM Multimedia Workshop on Multi-modal Misinformation Governance in the Era of Foundation Models · doi:10.1145/3689090.3689386 · Cited by 3 · see citing works
Nguyen-Le, H.-H., Tran, V.-T., Nguyen, D.-T., & Le-Khac, N.-A. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11608-4 · Cited by 1 · see citing works
Edge-Constrained Video Analytics: A Comprehensive Survey on Detection, Tracking, and Recognition
Toshpulatov, M. A., Lee, W., & Cho, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6614394
Birwadkar, R. (2025) · Journal of Computer and Forensic Sciences · doi:10.5937/jcfs4-64419
Protecting Language Models Against Unauthorized Distillation through Trace Rewriting
Ma, X., Yeoh, W., Zhang, N., & Vorobeychik, Y. (2026) · arXiv · doi:10.48550/arxiv.2602.15143
“Since its introduction, knowledge distillation has become a major workhorse in machine learning across a broad array of applications (Gou et al., 2021; Sanh et al., 2019), as it enables using large, complex, and expensive to execute teachers to train smaller and more efficient student models.”
How to Steal Reasoning Without Reasoning Traces
Zhang, T., Morris, J. X., & Shmatikov, V. (2026) · arXiv · doi:10.48550/arxiv.2603.07267
“…training, fine-tuning a model on step-by-step reasoning traces generated by a teacher model can transfer much of the latter’s reasoning capability, substantially outperforming fine-tuning on final answers alone (Hinton et al., 2015; Gou et al., 2021; Shridhar et al., 2023; Chen et al., 2025b).”
- Authors
- Gou, J., Yu, B., Maybank, S. J., & Tao, D.
- Affiliations
- Jiangsu University (CN)
- The University of Sydney (AU)
- University of London (GB)
- Corresponding author
- Tao, Dacheng
- Venue
- International Journal of Computer Vision
- Published
- 2021-03-22
- Volume
- 129
- Issue
- 6
- Pages
- 1789-1819
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W3034368386
- Semantic Scholar
- 1728cb805a9573b59330890ba9723e73d6c3c974
- Cited by
- 3,931
- References
- 256
- Citations by year
- 2026: 819 · 2025: 1078 · 2024: 868 · 2023: 693 · 2022: 330 · 2021: 135 · 2020: 8
- FWCI
- 182.16
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Journal of Computer Vision (publishedVersion)
- BIROn (Birkbeck, University of London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- knowledge distillation, model compression, model acceleration, teacher-student learning, knowledge categories, training program, resource-constrained deployment, deep neural network, computer vision
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
