The DOI matches its DataCite record; 9 of the 9 references it cites were found in a registry or an index.
Distilling the Knowledge in a Neural Network
Open accessAbstract
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“According to [19], both formulations should be equivalent in the limit of a high smoothing factor for logits (cf. § 3.2), but our choice of not smoothing leads to this different behavior.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These prior methods are highly effective on individual tasks, but they are not few-shot learners, require privileged access to datasets of human demonstrations or a target policy, and struggle to generalise to held-out tasks 25 .”
From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The key insight lies in the hypothesis that LLMs contain hidden knowledge (Hinton et al., 2015), latent representations within their weights that are not fully accessible through standard inference.”
From TT-SI: Self-Improving LLM Agents with Test-Time Training (2026) · cites this paper as Hinton et al., 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“KD technology originated from the concept of transferring knowledge from large models to small models, and was formally proposed by (Hinton et al. 2015) in 2015.”
From Survey: federated learning data security and privacy-preserving in edge-Internet of Things (2024) · cites this paper as Hinton et al. 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These challenges have motivated researchers to propose techniques for reducing transformers' size, latency and energy consumption for efficient inference for a wide range of applications.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [45]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“During training, finetuning a model on step-by-step reasoning traces generated by a teacher model can transfer much of the latter's reasoning capability, substantially outperforming fine-tuning on final answers alone (Hinton et al., 2015;Gou et al., 2021;Shridhar et al., 2023;Chen et al., 2025b).”
From How to Steal Reasoning Without Reasoning Traces (2026) · cites this paper as Hinton et al., 2015
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As noted in [19], knowledge distillation can also be expressed as a cross-entropy loss.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“We distill logits without any smoothing, unlike [19], because most of the proposals already produce a smooth distribution of scores. Moreover, in our case, both the old and the new networks are similar with almost the same parameters (in the beginning), and so smoothing the logits distribution is not necessary to stabilize the learning.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation, introduced by Hinton et al. [1], facilitates knowledge transfer from a large, cumbersome 'teacher' model to a smaller and more compact 'student' model.”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation provides a degree of controllable compression by allowing us to transfer knowledge from a larger model to a smaller one.”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Our approach provides a clear advantage by enabling a greater compression of the teacher model compared to the previous methods that relied on pruning [10-15], knowledge distillation (KD) [1,3,5,7,8], and the combination of both [16-20].”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation is another approach that involves training a smaller “student” model to replicate the behavior of a larger “teacher” model, requiring less computational power while retaining performance [161].”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [161]
The cited source says what the claim attributes to it.
In the citing paper
“In the context of action recognition (Tran et al. 2015; Wang et al. 2016; Carreira and Zisserman 2017; Feichtenhofer et al. 2019), there are handful of studies towards this end (Hoffman, Gupta, and Darrell 2016; Luo et al. 2018; Garcia, Morerio, and Murino 2018, 2019; Garcia et al. 2019; Stroud et al. 2020), all of which follow transfer learning scheme via knowledge distillation (Hinton et al. 201”
From Towards Good Practices for Missing Modality Robust Action Recognition (2023) · cites this paper as Hinton et al. 2015
The cited source says what the claim attributes to it.
In the citing paper
“Fine-Tuning (FT) based methods form a group which often uses a distillation [27] term to reduce catastrophic forgetting [44].”
From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Feature maps [8, 9, 10] and logits of a network [11, 12] are widely used as knowledge.”
From PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation (2021) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“Temperature scaling is commonly used in settings such as knowledge distillation (Hinton et al., 2015) and statistical mechanics (Jaynes, 1957).”
From On calibration of modern neural networks (2017) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Training on q(y | x; θ T ) is attractive since it gives more information about other classes for a given data point (e.g.similarity between classes) and has less variance in gradients (Hinton et al., 2015).”
From Sequence-Level Knowledge Distillation (2016) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006;Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT.”
From Sequence-Level Knowledge Distillation (2016) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Such a training improves the performance of the student model (alternatively, it can be regarded as a form of compression of the teacher model into a smaller one - the student).”
From Training data-efficient image transformers & distillation through attention (2020) · cites this paper as [24]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Distillation turns out to be a more effective method for obtaining interpretable models.”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [102]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our improvement compared to the implementation of [15] can be explained by the use of a larger projection head (3-layer, use of batchnormalizations and projection dimension of 256 ).”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We obtain 71 . 1% for MoCo after 300 epochs of training while [15] report 71 . 1% after 800 epochs of training.”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [15]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Self-citation(2014) · Cited by 33,419
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationEnsemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Self-citationAdaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Self-citationLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,202
Self-citationBuciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
Learning small-size DNN with output-distribution-based criteria
Li, J., Zhao, R., Huang, J.-T., & Gong, Y. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-432 · Cited by 252
Cited by
Sorted by citations
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“In addition, several works addressed the distillation of knowledge from an ensemble of models into a single model [80,289,290] .”
Metadata
- Authors
- Hinton, G., Vinyals, O., & Dean, J.
- ORCID
- Vinyals, Oriol
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1503.02531
- OpenAlex
- W1821462560
- Cited by
- 14,204
- References
- 9
- Citations by year
- 2026: 356 · 2025: 1286 · 2024: 2002 · 2023: 2430 · 2022: 1865 · 2021: 2483 · 2020: 1751 · 2019: 1033 · 2018: 555 · 2017: 270 · 2016: 115 · 2015: 51
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- knowledge distillation, neural network ensembles, model compression, ensemble averaging, mixture of experts, acoustic modeling, speech recognition, MNIST, deployment efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
