The DOI matches its DataCite record; 9 of the 9 references it cites were found in a registry or an index.
Distilling the Knowledge in a Neural Network
Open accessAbstract
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“According to [19], both formulations should be equivalent in the limit of a high smoothing factor for logits (cf. § 3.2), but our choice of not smoothing leads to this different behavior.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These prior methods are highly effective on individual tasks, but they are not few-shot learners, require privileged access to datasets of human demonstrations or a target policy, and struggle to generalise to held-out tasks 25 .”
From Learning few-shot imitation as cultural transmission (2023) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The key insight lies in the hypothesis that LLMs contain hidden knowledge (Hinton et al., 2015), latent representations within their weights that are not fully accessible through standard inference.”
From TT-SI: Self-Improving LLM Agents with Test-Time Training (2026) · cites this paper as Hinton et al., 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“KD technology originated from the concept of transferring knowledge from large models to small models, and was formally proposed by (Hinton et al. 2015) in 2015.”
From Survey: federated learning data security and privacy-preserving in edge-Internet of Things (2024) · cites this paper as Hinton et al. 2015
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These challenges have motivated researchers to propose techniques for reducing transformers' size, latency and energy consumption for efficient inference for a wide range of applications.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [45]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“During training, finetuning a model on step-by-step reasoning traces generated by a teacher model can transfer much of the latter's reasoning capability, substantially outperforming fine-tuning on final answers alone (Hinton et al., 2015;Gou et al., 2021;Shridhar et al., 2023;Chen et al., 2025b).”
From How to Steal Reasoning Without Reasoning Traces (2026) · cites this paper as Hinton et al., 2015
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As noted in [19], knowledge distillation can also be expressed as a cross-entropy loss.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“We distill logits without any smoothing, unlike [19], because most of the proposals already produce a smooth distribution of scores. Moreover, in our case, both the old and the new networks are similar with almost the same parameters (in the beginning), and so smoothing the logits distribution is not necessary to stabilize the learning.”
From Incremental Learning of Object Detectors without Catastrophic Forgetting (2017) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation, introduced by Hinton et al. [1], facilitates knowledge transfer from a large, cumbersome 'teacher' model to a smaller and more compact 'student' model.”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation provides a degree of controllable compression by allowing us to transfer knowledge from a larger model to a smaller one.”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Our approach provides a clear advantage by enabling a greater compression of the teacher model compared to the previous methods that relied on pruning [10-15], knowledge distillation (KD) [1,3,5,7,8], and the combination of both [16-20].”
From Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Knowledge distillation is another approach that involves training a smaller “student” model to replicate the behavior of a larger “teacher” model, requiring less computational power while retaining performance [161].”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [161]
The cited source says what the claim attributes to it.
In the citing paper
“In the context of action recognition (Tran et al. 2015; Wang et al. 2016; Carreira and Zisserman 2017; Feichtenhofer et al. 2019), there are handful of studies towards this end (Hoffman, Gupta, and Darrell 2016; Luo et al. 2018; Garcia, Morerio, and Murino 2018, 2019; Garcia et al. 2019; Stroud et al. 2020), all of which follow transfer learning scheme via knowledge distillation (Hinton et al. 201”
From Towards Good Practices for Missing Modality Robust Action Recognition (2023) · cites this paper as Hinton et al. 2015
The cited source says what the claim attributes to it.
In the citing paper
“Fine-Tuning (FT) based methods form a group which often uses a distillation [27] term to reduce catastrophic forgetting [44].”
From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“Feature maps [8, 9, 10] and logits of a network [11, 12] are widely used as knowledge.”
From PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation (2021) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“Temperature scaling is commonly used in settings such as knowledge distillation (Hinton et al., 2015) and statistical mechanics (Jaynes, 1957).”
From On calibration of modern neural networks (2017) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Training on q(y | x; θ T ) is attractive since it gives more information about other classes for a given data point (e.g.similarity between classes) and has less variance in gradients (Hinton et al., 2015).”
From Sequence-Level Knowledge Distillation (2016) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006;Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT.”
From Sequence-Level Knowledge Distillation (2016) · cites this paper as Hinton et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Such a training improves the performance of the student model (alternatively, it can be regarded as a form of compression of the teacher model into a smaller one - the student).”
From Training data-efficient image transformers & distillation through attention (2020) · cites this paper as [24]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Distillation turns out to be a more effective method for obtaining interpretable models.”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [102]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our improvement compared to the implementation of [15] can be explained by the use of a larger projection head (3-layer, use of batchnormalizations and projection dimension of 256 ).”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We obtain 71 . 1% for MoCo after 300 epochs of training while [15] report 71 . 1% after 800 epochs of training.”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [15]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Self-citation(2014) · Cited by 33,419
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationEnsemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Self-citationAdaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Self-citationLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,202
Self-citationBuciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
Learning small-size DNN with output-distribution-based criteria
Li, J., Zhao, R., Huang, J.-T., & Gong, Y. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-432 · Cited by 252
Cited by
Sorted by citations
Items 301–342 of 342
Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01939 · Cited by 14
“In this work, we use DAPO as the baseline for our RLVR experiments.”
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
Miech, A., Alayrac, J.-B., Laptev, I., Sivic, J., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00970 · Cited by 149
Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
Zhao, C., & Hospedales, T. (2020) · arXiv · doi:10.48550/arxiv.2012.04839 · Cited by 11
Toward Real-World Voice Disorder Classification
Kuo, H.-C., Hsieh, Y.-P., Tseng, H.-H., Wang, C.-T., Fang, S.-H., & Tsao, Y. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3270532 · Cited by 10
Visual information processing through the interplay between fine and coarse signal pathways
Zou, X., Ji, Z., Zhang, T., Huang, T., & Wu, S. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.07.048 · Cited by 9
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
Miller, E. (2024) · arXiv · doi:10.48550/arxiv.2411.00640 · Cited by 9
“It may be tempting to reduce the “sampling temperature”[10] of the model in order to reduce (or eliminate) the conditional variance.”
BackgroundA Survey on Self-Evolution of Large Language Models
Tao, Z., Lin, T.-E., Chen, X., Li, H., Wu, Y., Li, Y., Jin, Z., Huang, F., Tao, D., & Zhou, J. (2024) · arXiv · doi:10.48550/arxiv.2404.14387 · Cited by 9
Reasoning language models for more transparent prediction of suicide risk
McCoy, T. H., & Perlis, R. H. (2025) · BMJ Mental Health · doi:10.1136/bmjment-2025-301654 · Cited by 8
A Novel Mixed-Precision Quantization Approach for CNNs
Wu, D., Wang, Y., Fei, Y., & Gao, G. (2025) · IEEE Access · doi:10.1109/access.2025.3551802 · Cited by 8
Trusted Media Challenge Dataset and User Study
Chen, W., Chua, S. L. B., Winkler, S., & Ng, S.-K. (2022) · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · doi:10.1145/3511808.3557715 · Cited by 7
Generative AI-based data augmentation for improved bioacoustic classification in noisy environments
Gibbons, A., King, E., Donohue, I., & Parnell, A. (2024) · arXiv · doi:10.48550/arxiv.2412.01530 · Cited by 6
Transferring Dense Pose to Proximal Animal Classes
Sanakoyeu, A., Khalidov, V., McCarthy, M. S., Vedaldi, A., & Neverova, N. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00528 · Cited by 13
Assessing the State of Self-Supervised Human Activity Recognition using Wearables
Haresamudram, H., Essa, I., & Plötz, T. (2022) · arXiv · Cited by 107
Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
Du, D., Gong, G., & Chu, X. (2024) · arXiv · doi:10.48550/arxiv.2405.00314 · Cited by 4
“2) Distillation-assisted Training : Knowledge distillation (KD) [61] is a prominent methodology within the transfer learning domain, where knowledge is transferred from a more extensive “teacher” model to a more compact “student” model.”
BackgroundLLM2Swarm: Robot Swarms that Responsively Reason, Plan, and Collaborate through LLMs
Strobel, V., Dorigo, M., & Fritz, M. (2024) · arXiv · doi:10.48550/arxiv.2410.11387 · Cited by 22
“Scaling down hardware requirements and enabling on-board execution of LLMs on edge devices is a very active area of research; several approaches (e.g., finetuning [23], quantization [8; 29; 30], pruning [16; 31], distillation [22], and flash attention [7]) have been proposed for this.”
BackgroundASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
Yuan, Z., Shang, Y., Song, Y., Yang, D., Wu, Q., Yan, Y., & Sun, G. (2023) · arXiv · doi:10.48550/arxiv.2312.05821 · Cited by 4
“These methods primarily fall into three categories: weight quantization [Courbariaux et al., 2015, Dettmers et al., 2022], network pruning [LeCun et al., 1989, Fran-tar & Alistarh, 2023], and knowledge distillation [Hinton et al., 2015, Agarwal et al., 2023].”
MethodQuantification of Large Language Model Distillation
Lee, S., Zhou, J., Ao, C., Li, K., Du, X., He, S., Wu, H., Liu, T., Liu, J., Alinejad-Rokny, H., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.248 · Cited by 4
Domain-Adaptive Online Active Learning for Real-Time Intelligent Video Analytics on Edge Devices
Boldo, M., De Marchi, M., Martini, E., Aldegheri, S., & Bombieri, N. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3453188 · Cited by 4
“A promising approach is the knowledge distillation (KD) [11], which relies on predictions made by a heavier and more accurate model (referred to as the teacher ) to transfer knowledge to a light-weight model (referred to as the student ).”
BackgroundOnline continual streaming learning for embedded space applications
Mazouz, A. E., & Nguyen, V.-T. (2024) · Journal of Real-Time Image Processing · doi:10.1007/s11554-024-01438-4 · Cited by 3
Task-Specific Zero-Shot Quantization-Aware Training for Object Detection
Li, C., Chen, X., Wang, J., Zhao, K., & Chen, J. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02123 · Cited by 2
Islam, N., & Zulkernine, M. (2025) · arXiv · doi:10.48550/arxiv.2503.01089 · Cited by 9
“As IoV systems handle vast amounts of sensitive and private data, the need for PPML solutions has become increasingly critical on both the user and server sides.”
TT-SI: Self-Improving LLM Agents with Test-Time Training
Acikgoz, E. C., Qian, C., Ji, H., Hakkani-Tür, D., & Tur, G. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.462 · Cited by 1
“The key insight lies in the hypothesis that LLMs contain hidden knowledge (Hinton et al., 2015), latent representations within their weights that are not fully accessible through standard inference.”
Toward Scalable and Unified Example-Based Explanation and Outlier Detection
Chong, P., Cheung, N.-M., Elovici, Y., & Binder, A. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3127847 · Cited by 14
Personalized Negative Reservoir for Incremental Learning in Recommender Systems
Valkanas, A., Wang, Y., Zhang, Y., & Coates, M. (2024) · arXiv · doi:10.48550/arxiv.2403.03993 · Cited by 1
“where L BPR data batch and L 𝐾𝐷 teacher and student models, M 𝑇 and M 𝑆 KD incremental learning technique, L 𝐾𝐷”
Ye, J., Mao, Y., Song, J., Wang, X., Jin, C., & Song, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20219 · Cited by 1
Lightweight Identification of Malicious IoT Traffic via Cross-View Knowledge Distillation
Nie, F., Liu, W., Liu, G., Gao, B., Huang, J., & Yuen, C. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3596167 · Cited by 1
D-MEM: Dopamine-Gated Agentic Memory via Reward Prediction Error Routing
Song, Y., & Xin, Q. (2026) · arXiv · doi:10.48550/arxiv.2603.14597 · Cited by 1
“Knowledge distillation [Hinton et al., 2015] provides a natural pathway to eliminate this cost: a compact task-specific classifier trained on the soft outputs of the LLM-based Utility scorer could replicate its decisions at inference time with negligible latency.”
Vision-Guided Iterative Refinement for Frontend Code Generation
Sansford, H., Law, D. H. C., Liu, W., Tripathi, A., Agarwal, N., & Burg, G. J. J. V. D. (2026) · arXiv · doi:10.48550/arxiv.2604.05839 · Cited by 2
“Knowledge distillation was introduced as a student-teacher paradigm where a compact model is trained to match the softened output distributions of a more expressive teacher (Hinton et al., 2014).”
Explainable Anomaly Detection for System Logs in Distributed Environments
Gu, Z., Yue, W., & Liu, C. (2026) · Computers, Materials & Continua · doi:10.32604/cmc.2026.077388
“FedXLog employs knowledge distillation [46] technology to achieve model compression, designing a dual-model architecture.”
Self-Improving LLM Agents at Test-Time
Acikgoz, E. C., Qian, C., Ji, H., Hakkani-Tür, D., & Tur, G. (2025) · arXiv
“The key insight lies in the hypothesis that LLMs contain hidden knowledge (Hinton et al., 2015), latent representations within their weights that are not fully accessible through standard inference.”
Protecting Language Models Against Unauthorized Distillation through Trace Rewriting
Ma, X., Yeoh, W., Zhang, N., & Vorobeychik, Y. (2026) · arXiv · doi:10.48550/arxiv.2602.15143
“Knowledge distillation (KD) is a technique for transferring knowledge from a large, capable teacher model to a smaller, more efficient student model (Hinton et al., 2015).”
How to Steal Reasoning Without Reasoning Traces
Zhang, T., Morris, J. X., & Shmatikov, V. (2026) · arXiv · doi:10.48550/arxiv.2603.07267
“…training, fine-tuning a model on step-by-step reasoning traces generated by a teacher model can transfer much of the latter’s reasoning capability, substantially outperforming fine-tuning on final answers alone (Hinton et al., 2015; Gou et al., 2021; Shridhar et al., 2023; Chen et al., 2025b).”
The Limits of Query Monitoring against Distributed Model Extraction
Khilar, S. C. (2026) · arXiv
“That asymmetry makes distillation through the API attractive: collect the model’s answers to many well-chosen prompts and train a smaller model to imitate them (Hinton et al., 2015; Tramèr et al., 2016; Orekondy et al., 2019).”
Purified OPSD: On-Policy Self-Distillation Without Losing How to Think
Shen, Z., Tong, J., Yan, S., Shen, C., Chen, H., Ye, W., Hu, X., Miao, R., Wang, H., Zhao, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.02234
“Knowledge distillation [Hinton et al., 2015] transfers knowledge from a teacher to a student via soft label matching.”
SERA: Soft-Verified Efficient Repository Agents
Shen, E., Tormoen, D., Shah, S., Farhadi, A., & Dettmers, T. (2026) · arXiv · doi:10.48550/arxiv.2601.20789
Hollow-LLM Attack: Computationally Trivial Weights in Zero-Knowledge Verification of LLM Inference
Gong, C., Liu, B., & Li, M. (2026) · 2026 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp63933.2026.00258
Green AI: Balancing Model Complexity and Energy Footprint in Deep Learning
Aggarwal, D., Sharma, D., & Saxena, A. B. (2025) · 2025 3rd International Conference on Sustainable Computing and Data Communication Systems (ICSCDS) · doi:10.1109/icscds65426.2025.11167425
Cartridges: Lightweight and general-purpose long context representations via self-study
Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Tennien, W., Rudra, A., Zou, J., Mirhoseini, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.06266
Zweiger, A., Pari, J., Guo, H., Akyürek, E., Kim, Y., & Agrawal, P. (2025) · arXiv · Cited by 3
“In such a “teacher-student” formulation [71], a student model would be updated using edits proposed by a separate teacher model.”
LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Lee, B., Kim, D., You, Y., & Kim, Y. (2025) · arXiv · doi:10.48550/arxiv.2506.13771 · Cited by 7
“Training Details The LittleBit model parameters, initialized using the Dual-SVID method (Section 3.2), are optimized using QAT with knowledge distillation (KD) [63, 36, 64].”
InfluentialA Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
Belouadah, E., Popescu, A., & Kanellos, I. (2020) · arXiv · doi:10.48550/arxiv.2011.01844 · Cited by 235
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., Yu, F., Wang, W., & Gu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.05426 · Cited by 190
Metadata
- Authors
- Hinton, G., Vinyals, O., & Dean, J.
- ORCID
- Vinyals, Oriol
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1503.02531
- OpenAlex
- W1821462560
- Cited by
- 14,204
- References
- 9
- Citations by year
- 2026: 356 · 2025: 1286 · 2024: 2002 · 2023: 2430 · 2022: 1865 · 2021: 2483 · 2020: 1751 · 2019: 1033 · 2018: 555 · 2017: 270 · 2016: 115 · 2015: 51
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- knowledge distillation, neural network ensembles, model compression, ensemble averaging, mixture of experts, acoustic modeling, speech recognition, MNIST, deployment efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
